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PREFACE 


This  report  addresses  four  questions  pertaining  to  the  allocation 
of  funds  for  research  and  development! 

1.  How  can  we  determine  future  military  deficiencies  for  which 
technological  advancements  may  provide  remedies? 

2.  What  tools  can  be  developed  to  relate  these  deficiencies  to 
the  range  of  alternative  technological  remedies? 

3.  What  procedures  can  be  used  to  assess  and  compare  the  various 
remedies  for  their  potential  for  technological  advancement? 

4.  How  could  the  information  developed  in  (1),  (2),  and  (3)  be 
used  for  management  purposes? 

Various  research  strategies  are  identified  in  pursuit  of  answers  to 
these  questions.  In  the  context  of  the  third  question,  analytical 
techniques  are  discussed  which  appear  promising  for  quantifying  tech- 
nological  advancement. 

The  mathematical  Appendix  to  this  report  first  appeared  in  Rand 
Report  R-1306-EDA/NIH,  Univariate  and  Multivariate  Log-Linear  and 
Logistic  Models » by  Marc  Nerlove  and  S.  James  Press,  December  1973. 

This  discussion  of  the  issues  is  intended  as  a guide  to  approaches 
to  program  planning  by  Department  of  Defense  groups  undertaking  or 
funding  exploratory  R&D. 


SUMMARY 


Itiis  report  is  concerned  with  providing  a policymaker  with  infor- 
mation that  will  help  him  in  his  choice  of  exploratory  R&D  options — 
options  that  will  subsequently  affect  the  feasibility  or  availability 
of  new  systems.  The  fundamental  issue  is.  How  do  we  assess  the  rela- 
tive degree  of  technological  advancement  of  different  types  of  projects 
(such  as  those  associated  with  enhancements  of  aircraft  and  missile 
capabilities)  that  may  require  long  periods  of  research  and  develop- 
ment, if  they  are  feasible  at  all? 

In  this  study  we  approach  the  problem  by  first  relating  the  assess- 
ment of  technological  advancement  to  R&D  program  management  objectives. 
Broadly  defined,  program  management,  as  practiced  by  the  major  federal 
R&D  funding  agencies,  consists  of  four  interacting  activities:  planning, 

development,  evaluation,  and  utilization.  Using  these  activities  as 
a framework,  it  is  possible  to  determine  areas  in  which  the  assessment 
of  technological  advancement  may  make  a difference,  the  degree  depend- 
ing on  whether  we  are  considering  a long-term  or  short-term  horizon. 

In  long-term  program  planning,  for  instance,  assessment  of  technological 
advancement  can  be  very  useful  in  gathering  information  on  a technology 
or  program  objective  in  order  to  identify  important  new  prospects  for 
R&D,  and  to  indicate  what  resources  will  be  required  to  pursue  them. 

Since  technological-advance  assessments  are  obviously  important 
to  program  planning  and  development,  how  can  such  assessment  be  ob- 
tained. The  first  step  is  to  determine  the  scope  of  the  assessment 
problem  by  identifying  which  potential  advances  will  be  useful  and 
then  considering  different  technologies  that  may  be  alternative  solu- 
tions to  the  same  problems.  Once  we  have  established  what  our  assess- 
ment objectives  are,  we  can  develop  criteria  for  deciding  who  is  quali- 
fied to  make  appraisals  of  the  potential  advances  in  those  objectives, 
and  then  design  procedures  for  obtaining  and  grouping  expert  judgments. 

Basic  to  our  approach  to  technological-advance  assessment  is  the 
notion  that  expertise  does  exist,  that  there  are  people  who  have  con- 
siderable knowledge  and  understanding  of  the  mechanisms  underlying 
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particular  problems  and  thus  can  do  an  appreciably  better  job  of  fore- 
casting long-term  trends  and  changes  than  the  non-expert.  But  how  do 
we  recognize  an  expert?  How  many  experts  constitute  a "good"  panel? 
What  approach  should  be  used  in  eliciting  their  responses?  What  types 
of  questions  should  be  asked,  and  how  should  they  be  structured?  This 
study  addresses  these  and  other  important  issues  associated  with  re- 
solving the  procedural  and  administrative  problems  involved  in  select- 
ing a panel  of  experts,  in  eliciting  informed  judgments  about  the 
degree  of  technological  advance  or  relevant  projects,  and  in  designing 
a survey  questionnaire  for  measuring  those  judgments. 

Three  methods  of  multivariate  analysis  are  described  for  quantify- 
ing and  analyzing  group  judgment  data  collected  from  a panel  of  experts. 
Those  of  the  methods  that  are  known  in  earlier  literature  include  multi- 
dimensional scaling  of  individual  differences,  and  subjective  probabil- 
ity procedures,  including  the  Bayesian  approach.  A new  procedure, 
which  was  developed  specifically  for  this  application,  involves  the 
use  of  simultaneous  equation  system  models  in  which  the  response  (de- 
pendent) variables  are  categorical,  i.e.,  discrete  (possibly  having 
a value  of  zero  or  one,  oply)  and  unordered.  Such  dependent  variables 
are  interpretable  in  terms  of  probabilities.  Since  project  feasibility 
can  be  measured  by  the  probability  that  a new  capability  can  be  de- 
veloped, and  this  probability  can  be  related  to  other  probabilities 
that  various  components  can  be  developed,  the  new  procedure  described 
in  the  report  has  great  potential  applicability.  (The  details  of  this 
new  statistical  procedure  are  provided  in  an  appendix.) 

Further  research  is  still  needed  in  a number  of  areas.  For  ex- 
ample: (1)  The  methods  for  selecting  a panel  of  experts  should  be 

refined.  (2)  Planning  objectives  should  be  devised  for  selecting 
technology  areas.  (3)  Criteria  should  be  established  for  determining 
the  number  and  type  of  experts.  (4)  An  interrogation  procedure  should 
be  developed,  with  the  questions  designed  to  assess  technological  ad- 
vancement in  specific  technologies.  Also,  at  least  two  types  of  anal- 
ysis should  be  undertaken:  individual-differences  scaling  using  a 

mu^t:^stage  controlled  feedback  approach,  and  a categorical-dependent- 
variable  multivariate  regression  (but  without  the  feedback  data) . 


-vii- 


In  short,  the  material  presented  in  this  report  is  as  much  proposal 

for  the  initiation  of  a technological-advance  assessment  based  on  the 

principles  and  procedures  described  herein  as  it  is  a presentation  of 
research  findings. 
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I.  INTRODUCTION 


This  report  presents  techniques  and  approaches  for  assessing  the 
feasibility  of  new  systems  or  projects.  We  discuss  ways  of  generating 
information  that  will  help  us  to  better  articulate  a "menu"  of  R&D 
options,  and  ways  of  developing  analytical  tools  for  refining  the  com- 
parison among  options.  We  also  show  how  technical  and  economic  exper- 
tise can  be  used:  for  assessing  technological  advances  and  trends, 

using  quantitative  techniques;  for  comparing  technological  advances 
for  various  projects;  and  for  determining  how  technological  advances 
affect  R&D  costs. 

Because  there  are  few  inevitable  or  unalterable  technological 
trends,  and  few  fixed  intervals  between  experimental  and  operational 
hardware,  it  is  quite  important  that  we  improve  our  capability  for 
assessing  technological-advance  prospects  in  order  to  improve  planning 
and  management  activities  for  the  more  heterogeneous  R&D  projects.  That 
is,  managers  should  influence  technological  advance  through  deliberate 
action. 

In  this  report,  we  propose  a methodology  that  will  help  to  deter- 
mine the  allocation  of  R&D  and  thus  improve  the  votes  of  technological 
advance  and  veduce  the  feasible  leadtime  from  experimental  to  opera- 
tional capabilities.  To  achieve  these  goals,  we  must  have  a thorough 
understanding  of  (1)  the  "value"  of  a successful  technological  advance 
of  any  objective,  (2)  the  scope  of  the  objectives  that  determine  which 
technologies  should  and  should  not  be  considered  as  alternatives,  (3) 
the  procedures  for  assessing  and  comparing  technological-advance  al- 
ternatives, and  (4)  the  management  activities  into  which  the 
technological-advance  assessment  results  will  fit.  Here  we  focus  most 
heavily  on  topics  two  and  three — the  use  of  broad  objectives  in  develop- 
ing methods  for  acquiring  and  analyzing  information  with  which  to  com- 
pare the  feasibility  of  various  technological  advances.  Topics  one 
and  four  must  be  considered,  however,  if  the  end  results  of  the  assess- 
ment are  to  be  useful. 

Past  research  at  Rand  has  yielded  methodologies  for  assessments 
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of  technological  advance  that  concentrated  on  "outputs"  of  R&D  leading 
to  improve  military  capabilities  (as  a measure  of  technological 
success  ) , rather  than  on  specific  technologies  (as  "inputs"  to  im- 
prove capability)  that  may  or  may  not  be  economically  efficient  to 
pursue.  For  example,  RandTs  turbine  engine  research  concentrated  on 
an  assessment  of  technological  trends  based  on  performance  advancement 
rather  than  on  the  specific  considerations  of  compressor  designs  or 
use  of  advanced  alloys.  Maintaining  such  trends  may  call  for  different 
"input-technology"  solutions  at  different  times,  and  trying  to  "push" 
ahead  of  the  trend  may  become  very  costly  or  risky.  The  resulting 
trend  line  could  be  characterized  as  one  of  technology  embodied  in 
operational  hardware.  Conceptually,  one  can  imagine  such  trends  as 
extending  from  the  advancement  of  scientific  knowledge  (as  it  applies 
to  improved  military  capability)  to  the  varying  degrees  of  embodiment 
in  experimental  breadboard  or  brassboard,  prototype,  or  production 
hardware.  Although  the  analytical  methods  of  technological-advance 
assessment  thus  far  developed  are  appropriate  to  R&D  projects  that 
lead  to  equipment  with  the  same  set  of  operational  attributes  over  a 
long  period  of  time,  these  methods  are  not  directly  appropriate  to 
more  heterogeneous  R&D  projects  and  component  advancements. 

To  develop  sound  methods  for  such  applications,  we  have  looked 
for  the  best  way  of  using  historical  data  and  subjective  expertise 
in  assessing  the  advancement  attainable  in  future  projects.  If  we 
use  a panel  of  "experts"  in  relevant  fields,  there  are  a number  of 
important  factors  that  must  be  considered.  We  must  first  determine 
who  should  participate  in  an  exercise  to  evaluate  alternative  tech- 
nologies, and  then  devise  procedures  to  formulate  carefully  a broad 
range  of  questions  with  respect  to  technologies,  resource  requirements, 
and  time  horizons  that  would  be  required  to  make  relative  comparisons 


Previous  work  in  these  general  areas  include  A.  J.  Alexander 
and  J.  R.  Nelson,  Measuring  Technological  Change:  Aircraft  Turbine 

Engines , R-1017-ARPA/PR,  June  1972;  A.  J.  Harman,  Choice  Among  Stra- 
tegies for  System  Acquisition , The  Rand  Corporation,  P-4794,  March 
1972;  J,  R,  Nelson  and  F.  S.  Timson,  Relating  Technology  to  Acquisi- 
tion Cost:  Aircraft  Turbine  Engines , The  Rand  Corporation,  R-1288-PR, 

March  1974. 
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across  a technological  area.  We  must  also  give  serious  consideration 
to  the  kinds  of  incentives  that  should  be  provided  to  the  participants , 
so  that  they  will  take  the  exercise  seriously  and  provide  thoughtful, 
conscientious  responses  based  on  their  expertise.  Procedures  for 
eliciting  expert  judgments  are  discussed  in  Section  III.  For  example, 
one  way  of  eliciting  information  would  be  to  request  it  in  terms  of 
alternative  "paths"  to  the  planning  objective,  at  the  same  time  obtain- 
ing the  probabilities  of  success  of  each  node  in  the  path  and  the 
plausible  range  of  time  and  resource  commitments  needed  for  a reason- 
able probability  of  success.  Such  information  would  provide  valuable 
insights  into  promising  technological  areas  for  R&D  support. 

Once  we  have  elicited  information  from  a panel  of  experts,  we  will 
need  valid,  appropriate  methods  of  analyzing  this  information.  The  raw 
information  may  be  of  direct  interest  to  the  extent  that  detailed  re- 
sponses are  obtained  from  some  of  the  experts.  It  may  be  desirable  to 
make  careful  engineering  evaluations  of  the  responses  of  the  group  and 
to  use  in-house  judgments  to  provide  decision  alternatives.  But  it 
also  seems  quite  likely  that  insights  can  be  obtained  by  using  more 
formal  statistical  procedures  for  assembling  the  group  judgments.  Such 
procedures  are  proposed  in  Section  IV. 

Before  we  can  discuss  technological  assessment  techniques  in  de- 
tail, however,  we  need  to  specify  which  phase  of  the  planning  or  opera- 
tion of  an  R&D  program  we  are  addressing.  We  assume  that  we  are  at  a 
very  early  phase  of  planning  a new  set  of  research  projects . Therefore, 
techniques  of  analysis  should  yield  a broader  but  less  thorough  under- 
standing of  individual  project  areas  than  those  for  project  selection 
or  evaluation.  This  is  the  subject  of  Section  II. 

We  are  attempting  to  build  on  our  past  experience  in  developing 
techniques  for  assessing  technological  advance.  Such  assessments  may 
provide  important  information  for  decisionmaking,  especially  if  we  con- 
sider the  prospects  for  advance  in  the  context  of  the  resources  needed 


The  kinds  of  persons  to  be  involved  in  any  one  technological  as- 
sessment will  clearly  depend  on  the  scope  of  the  technologies  involved; 
but  in  most  cases  a broad  range  of  perspectives,  from  both  industrial 
and  university  backgrounds,  may  be  desirable. 
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to  support  the  effort.  But  other  information  is  desirable:  Is  the 
subject  of  R&D  being  ignored  completely  by  other  supporters  (in  the 
DoD  and  other  federal  agencies)  and  by  private  industry?  What  are  our 
allies  doing  in  the  area?  The  Soviets?  What  are  their  future  plans? 
Thus  the  assessments  and  related  policy  issues  need  to  be  addressed  in 
terms  of  specific  technologies,  and  it  is  this  aspect  of  the  decision 
process  on  which  we  will  focus. 
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II,  PROBLEM  DEFINITION:  MANAGEMENT  OBJECTIVES 

IN  MEASURING  TECHNOLOGICAL  ADVANCEMENT 


MANAGEMENT  FRAMEWORKS3 

We  can  rather  broadly  characterize  a management  function  as  con- 
sisting  of  four  major  interacting  parts:  (1)  planning  activities;  (2) 

the  initiation,  selection,  and  development  of  project  ideas  consistent 
with  planning  outcomes;  (3)  the  evaluation  of  project  results  and  the 
redirection  of  program  plans;  and  (4)  efforts  to  ensure  that  project 
results  are  used,  with  the  consequent  effect  of  improving  military  ca- 
pability. (See  Table  1.)  These  activities  do  not  necessarily  proceed 
sequentially,  even  for  any  one  program  area,  but  are  often  parallel 
efforts,  highly  interactive  and  oriented  toward  future  conditions.  In 
a sense,  the  program  planning  activities  are  efforts  to  anticipate  where 
operational  hardware  will  (or  should)  be  in  terms  of  advanced  capabil- 
ity at  some  future  period  (e.g.,  10  or  15  years  hence),  while  the  pro- 
gram development  and  evaluation  activities  are  designed  to  see  that 
the  technology  base  moves  forward  at  a sufficient  pace  to  achieve  this 
advanced  capability.  Finally,  program  use — or  technology  transfer  for 
implementation  in  weapon  systems — must  be  effected  to  make  use  of  these 
improved  (potential)  military  capabilities. 

Program  planning  is  accomplished  in  quite  different  ways  in  dif- 
ferent federal  R&D  funding  agencies  (see  Table  2).  For  example,  the 
Goddard  Space  Flight  Center  develops  very  specific  objectives  in  terms 
of  building  operational  capabilities  for  the  NASA  missions;  whereas  in 
the  National  Institutes  of  Health,  and  in  various  research  divisions 
of  the  National  Science  Foundation,  program  objectives  concern  the  ad- 
vancement of  science  in  well-defined  disciplines,  often  broken  down 
into  fields  of  inquiry.  The  latter  are  quite  unlike  the  objectives 
that  the  Defense  Advanced  Research  Projects  Agency  (ARPA)  emphasizes 
for  R&D  programs  for  enhancing  military  capability. 


The  subject  matter  addressed  here  draws  heavily  on  the  research 
undertaken  by  J.  G.  Wirt,  A.  Lieberman,  and  R.  E.  Levien,  R&D  Manage- 
ment: Methods  Used  by  Federal  Agencies , The  Rand  Corporation,  R-1156- 

HEW,  January  1974. 
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Table  1 

A PROGRAM  MANAGEMENT  FRAMEWORK 


1.  Program  Planning 

a.  Identifying  future  military  and  technological 

deficiencies . 

b.  Assigning  priorities  to  R&D  objectives. 

c.  Assessing  problem  scope,  difficulty,  cost, 

and  avenues  for  solution. 

d.  Preparing  plans  for  analyses  and  program 

development . 

2.  Program  Development 

a.  Generation  of  projects — analysis  of  program 

needs  and  solicitation  of  ideas  for 
relevant  resources. 

b.  Selection  of  projects — evaluation  of  solicited 

and  unsolicited  proposals  and  allocation  of 
R&D  resources. 

c.  Monitoring  of  projects — communication  and 

direction  to  the  research  community 
relative  to  current  program  plans. 

d.  Evaluation  of  projects — assessment,  both  cur- 

rent  and  in  retrospect. 

3.  Program  Evaluation 
(or  Redirection) 

a.  Recurrent  assessments  of  relevance,  feasibilty, 

and  cost/benefit  throughout  program  life 
cycle. 

b.  Plans  for  changes  in  response  to  realized  and 

unmet  progress. 

c.  Tests  of  effects  and  cost  following  completion. 

d.  Feedback  of  results  to  planning  activities. 

4.  Program  Utilization 

(or  Technology  Transfer) 

a.  Incorporating  users1  preferences  in  planning 

and  design. 

b.  Integrating  the  utilization  and  development 

processes  through  communication,  collocation, 
and  shared  responsibility. 

Table  2 


EXAMPLES  OF  R&D  MANAGEMENT  PRACTICES 


Program 

Agency 

Planning 

Development 

Evaluation 

Utilization 

National  Aeronautics  and  Space 
Administration: 

Goddard  Space  Flight  Center 

Highly  directed 
with  specific 
objectives 

Reliance  on  internal 
staff  for  project 
selection 
Close  monitoring 

Regularly  and  fre- 
quently evaluated 

Plan  for  utilization 
included  in  projects 

National  Science  Foundation: 
Research  Applied  to  National 
Needs  (RANN) 

Engineering,  Mathematics, 
Physics  Divisions 

General  objectives 
and  priorities 

Project  ideas  generated 
externally 

Consultation  with  re- 
search community  on 
project  selection 
Partial  monitoring 

Regularly  evaluated, 
but  infrequently 

Plan  for  utilization 
Included  in  projects 

By  disciplines  or 
fields  of  inquiry 

Support  of  individual 
scientists 

External  mail  review 
with  feedback  to 
reviewers 

: Little  monitoring 

Program  manager  de- 
fends his  project 
selection  decisions 

Little  direct  attention 

Biological  and  Social  Sciences 

By  disciplines 
(within  these, 
by  lines  of  in- 
quiry) 

Support  of  individual 
scientists 

External  panel  review  of 
borderline  decisions 
Little  monitoring 

Evaluation  implicit  in 
considering  renewed 
or  continued  support 

Little  direct  attention 

National  Institutes  of  Health 

By  disciplines, 
collections  of 
similar  subjects 

Support  of  individual 
scientists 

Unsolicited  proposals 
Dual  panel  review 

(greatest  attention  to 
evaluating  proposals) 
Little  monitoring 

Evaluation  implicit  in 
considering  renewed 
or  continued  support 

Little  direct  attention 

Office  of  Naval  Research 

By  disciplines  and 
directed  at  at- 
tracting most  ca- 
pable researchers 

Seek  new  researchers 
No  formal  review  of 
project  Ideas 
Monitor  activities  (once 
or  twice  a year) 

Evaluation  implicit  in 
considering  renewed 
or  continued  support 

A part  of  the  program 
manager's  mandate 

SOURCE:  Abstracted  from  research  material  developed  through  extensive  interviewing  by  A.  Lieberman  and  J.  G.  Wirt.  See 

also  J.  G.  Wirt,  A.  Lieberman,  and  R.  E.  Levien,  R&D  Management:  Methods  Used  by  Federal  Agencies , Tne  Rand  Corporation 

R-1156-HEW,  January  1974. 
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There  is  also  a broad  range  of  techniques  for  developing  R&D  pro- 
grams once  plans  have  been  articulated.  Many  of  these  rely  on  outside 
experts  to  evaluate  proposals  or  to  make  judgments  of  one  sort  or  an- 
other. For  example,  the  NSF  engineering,  mathematics,  and  physics 
divisions  rely  on  mail  surveys;  i.e.,  proposals  submitted  by  scientists 
are  sent  out  to  independent  experts  with  a request  for  a written  ap- 
praisal of  the  proposals1  scientific  merit.  Program  directors  then 
have  the  option  of  feeding  back,  to  the  same  group  of  appraisers,  the 
combined  judgments  to  obtain  further  reactions.  (Extensions  of  this 
concept  are  discussed  in  detail  in  Sections  III  and  IV.)  As  for  the 
evaluation  and  use  of  the  various  research  programs,  again  practices 
vary;  but  often  too  little  attention  is  paid  during  development  to  the 
ultimate  application  of  results. 

UTILITY  OF  TECHNOLOGICAL-ADVANCE  ASSESSMENT 

To  further  clarify  areas  where  technological-advance  assessment 
may  make  a difference,  it  is  important  for  us  to  refine  further  what 
we  mean  by  program  "planning  and  development  activities.  For  example, 
some  technological  assessment  techniques  are  better  for  planning  for 
the  reasonably  short-term  horizon  or  for  evaluating  projects  for  which 
proposals  have  already  been  submitted,^  while  others  have  been  designed 
for  the  longer-term  horizon. 

Long-term  program  planning  is  particularly  difficult.  Technological- 
advance  assessment  may  be  very  useful  for  this  purpose — in  gathering 
information  (perhaps  by  making  use  of  outside  expertise)  on  a technology 
or  program  objective  in  order  to  identify  important  new  prospects  for 
R&D.  These  assessments  may  also  indicate  what  resources  will  be  re- 
quired to  pursue  these  new  prospects.  Such  indications  would  be  help- 
ful in  moderate-time-horizon  planning,  since  they  would  provide  guid- 
ance on  the  kinds  of  Requests  for  Proposals  to  issue  and  would  focus 
very  detailed  evaluations  of  project  ideas  onto  subjects  having 

^Evaluations  at  this  stage  of  decisionmaking  may  best  be  conducted 
on  the  basis  of  scientific  merit,  novelty  of  approach,  relevance  to 
program  objectives,  and  so  forth. 
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potentially  high  payoffs.  Early  planning  exercises  may  also  make  it 
easier  to  compare  the  applicability  of  R&D  ideas  under  consideration 
for  a variety  of  technologies,  and  the  research  community  may  be  en- 
couraged to  propose  projects  for  funding  that  the  technology-advance 
assessment  exercises  reveal  as  most  promising. 

Since  technological-advance  assessments  would  obviously  be  useful 
to  R&D  planning  and  program  development,  how  can  such  assessments  be 
obtained?  There  are  a number  of  issues  that  we  must  look  at  in  some 
detail  in  deciding  how  to  go  about  a technological-advance  assessment. 
How  do  we  identify  the  appropriate  scope  of  our  assessment  problem? 

It  is  highly  desirable  to  identify  potential  advances  that  will  be 
militarily  useful  and  also  to  consider  different  technologies  that  may 
be  alternative  solutions  to  the  same  problems.  Once  we  have  determined 
what  our  assessment  objectives  are,  we  ought  to  have  some  criteria  for 
deciding  who  is  qualified  to  make  appraisals  of  potential  advances  in 
those  objectives.  Also,  we  must  seriously  consider  the  problem  that 
some  of  the  participants  in  an  exercise  may  tend  not  to  respond  seri- 
ously and  fully.  In  the  early  planning  stages  especially,  discussion 
should  not  be  at  a level  of  detail  that  would  pose  a threat  of  reveal- 
ing proprietary  information  to  competitors.  Finally,  there  are  a num- 
ber of  questions  of  how  best  to  use  the  various  opinions  that  are  ob- 
tained: how  to  condense  them,  analyze  them,  and  input  them  to  the 

decisionmaking  process.  These  subjects  are  discussed  in  detail  in  Sec- 
tion III. 

The  concept  of  developing  a technological  base  that  will  later  be 
of  use  to  advance  the  capability  of  operational  military  hardware  rests 
on  the  premise  that  we  can  discern  the  potential  military  "worth"  of 
future  technological  advances;  this  is  ultimately  the  responsibility 
of  senior  DoD  officials. 

One  approach  to  evaluating  military  worth  is  to  develop  scenarios 
for  future  time  periods  to  be  used  in  identifying  the  technological 
"weak  links"  in  the  systems  our  forces  will  need.  Such  a study  should 

"top-down"  planning  of  technological  advancement  was  recommended 
by  Dr.  John  S.  Foster,  Jr.,  in  his  keynote  address  for  the  national 
security  session  of  the  American  Institute  of  Aeronautics  and  Astronau- 
tics convention,  January  10,  1973. 
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identify  a set  of  steps  or  decision  points  to  consider  in  deciding 
whether  advances  in  a given  technological  area  would  be  worthwhile. 

This  approach  might  be  characterized  as  an  effort  directed  at  deter- 
mining what  a military  service  needs. 

Another  approach  would  be  to  survey  the  many  different  kinds  of 
Required  Operational  Capabilities  (ROCs)  as  identified  by  the  individ- 
ual services.  To  the  extent  that  the  underlying  problems  are  similar, 
research  and  exploratory  development  may  be  warranted.  The  initiation 
of  ARPA's  human  resources  research  could  be  viewed  as  arising  out  of 
such  a perception.  This  approach  would  address  the  "use"  issue,  in 
that  it  would  be  likely  to  yield  results  the  services  would  implement . 

STRUCTURING  A TECHNOLOGICAL- ADVANCE  ASSESSMENT 

A way  in  which  technologies  can  be  viewed  as  alternative  solutions 
to  the  same  future  military  objective  is  shown  graphically  in  Fig.  1. 
After  first  looking  at  very  broad  planning  objectives,  we  can  identify 
the  various  technologies  that  contribute  to  these  objectives.  From 
these,  we  can  determine  what  projects  have  been  done  in  the  past  in 
each  technological  area  and  what  future  projects  seem  to  be  best  fitted 
to  advance  the  capabilities  (or  remedy  the  deficiencies)  for  the  given 
objective. ^ The  number  of  potential  projects  yielding  some  advance- 
ment of  the  overall  planning  objective  could  be  quite  large,  both  be- 
cause of  the  number  of  technologies  that  contribute  to  any  one  objec- 
tive and  because  of  the  number  of  options  within  a technological  area. 

Statistical  procedures  and  engineering  analysis  may  both  yield 
useful  (and  different)  "figures  of  merit"  for  planning  purposes.  Anal- 
ysis of  aircraft  turbine  engines,  for  example,  has  yielded  an  equation 
interpretable  in  terms  of  a tradeoff  surface  among  the  desirable, 

7 

This  description  is  rather  simplified,  especially  because  it  is 
highly  desirable  that  the  activities  depicted  in  Fig.  1 interact  with 
one  another.  Thus  a first  cut  at  the  planning  objectives  may  identify 
promising  projects  for  funding;  the  results  of  these  projects  will 
provide  valuable  information  for  a second  cut  at  the  objectives,  as 
well  as  an  expanded  technology  base  for  possible  utilization. 

^Also,  it  is  quite  possible  that  a given  project  might  benefit 
several  objectives. 


Planning  objectives 
(tactical  domain): 


Technologies:  ^ 
Projects:  ♦ 


i 


i 


Surveillance  and 
weapons  delivery 

• RF  and  laser  radar 

• ELINT 

• ECM 

• Passive  sensors 

• Sonar 


Control  Systems 


Detectors 

Signal  Processing 

Display 

Packaging 

Ancillary  Power 

Synthesis 

• Materials 

• Analog 

• Liquid  crystals 

• Environmental 

# 

• 

• Cryogenics 

• Digital 

• Light-emitting 

• Weight 

• • 

• 

• Component 

diodes 

• Volume 

• 

• 

packaging 


• TV 


Fig  . 1 — Structure  for  assessment  problem 
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user-oriented  performance  dimensions.  This  tradeoff  function  captures, 
to  some  extent,  both  the  feasible  tradeoffs  among  performance  dimensions 
from  the  standpoint  of  the  underlying  laws  of  physics  and  engineering- 
design  considerations,  and  the  degree  of  relative  emphasis  on  the  vari- 
ous performance  dimensions  that  the  users  have  sought.  The  overall 
equation  is  as  follows : 


_ fI  . „ Tempa  Thrust^  Q max^ 

Engine  "vintage"  = K K T > 

Weight  SFCn 


in  which  the  engine  "vintage"  is  a surrogate  for  the  relative  advance- 
ment of  the  various  engines  observed  in  past  development  activities, 
and  the  variables  on  the  right  are  (in  the  numerator)  the  turbine  inlet 
temperatures,  the  maximum  thrust,  and  the  pressure  ratio;  and  the  en- 

g 

gine's  weight  and  specific  fuel  consumption  (in  the  denominator).  It 
should  be  emphasized  that  this  kind  of  statistically  derived  relation- 
ship, which  reflects  in  part  the  users’  demands,  is  only  appropriate 
for  selecting  new  projects  involving  modest  variations  in  each  of  these 
individual  performance  dimensions.  For  example,  engines  are  not  neces- 
sarily "superior"  as  we  push  for  extreme  reductions  in  specific  fuel 

9 

consumption,  holding  all  other  variables  constant. 

Let  us  consider  surveillance  sensors  for  an  illustration  of  the 
derivation,  from  engineering  analysis,  of  a broadly  based  "figure  of 
merit"  that  could  be  used  to  determine  what  technological  areas  we 
would  want  to  include  within  a technological-advance  assessment.  Such 


8 

For  further  details  of  this  analysis,  see  Alexander  and  Nelson, 
op.  cit.  In  the  equation  as  actually  estimated  statistically,  the  en- 
gine "vintage"  was  captured  by  the  time  (in  quarters  since  1942)  at 
which  the  engine  passed  its  model  qualification  test.  The  various 
parameters  were  estimated  as  follows:  K = -1187,  a = 156,  6 = 18.8, 

y = 11.7,  a = 26.5,  and  r\  = 20.6. 

9 

In  fact,  the  estimated  parameters  would  be  misleading  in  any  case 
for  such  combinations  of  variable  values  well  beyond  the  range  of  the 
variables  in  the  original  data. 
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a figure  of  merit  was  constructed  as  a by-product  of  another  recent 
research  study . ^ In  equation  form,  it  is: 


in  which  the  figure  of  merit,  cp,  is  the  product  of  a measure  of  the 

2 

information  rate,  VW/AS  , and  the  square  of  the  standoff  range,  R. 

In  this  information-rate  term,  V stands  for  the  attainable  velocity 
at  which  the  sensor  moves  and  W stands  for  the  swathe  -width;  together 
the  terms  in  the  numerator  are  the  measure  of  the  area  coverage  rate. 

A is  the  intensity  resolution  of  the  system  and  6 is  the  spatial  reso- 
lution. One  would  also  want  to  build  into  such  a figure  of  merit 
the  reliability  of  the  system  and  its  potential  number  of  hours  of 
operation  in  some  standard  calendar  period  (as  a measure  of  the  extent 
of  all-weather  and  day-night  capabilities) . Such  a figure  of  merit 
naturally  leads  to  a broad  investigation  of  appropriate  technological 
advancement  objectives;  for  example,  V in  part  captures  the  data- 
processing  rate  of  the  surveillance  sensor,  and  6 is  determined  by  the 
capabilities  of  the  front  end  of  the  sensor. 

Such  figures  of  merit  could  potentially  be  developed  for  many  dif- 
ferent military  capabilities. 

■^See  H.  H.  Bailey,  Side -Looking  Radar  (U) , R-581/3-ARPA,  The 
Rand  Corporation,  June  1971  (Confidential). 

^For  further  detailed  description  of  these  terms,  see  ibid., 
pp.  3-3. 
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III.  COLLECTION  OF  GROUP  JUDGMENT  DATA; 
SURVEY  DESIGN  CONSIDERATIONS 


In  this  section  we  will  focus  on  the  problems  encountered  in  de- 
signing procedures  for  eliciting  and  grouping  expert  judgments.  We 
believe  that  by  eliciting  the  judgments  of  experts  and  studying  their 
views  on  the  relevant  issues  in  a formal  way  (without  necessarily  look- 
ing for  consensus) , we  will  be  able  to  improve  on  existing  criteria 

for  budget  allocation  that  currently  rely  on  a somewhat  less  formal 

, J , 12 
methodology. 

The  basis  for  our  approach  to  assessing  technological  advance 
rests  with  the  notion  that  there  is  such  a thing  as  expertise,  and  that 
many  experts  are  better  than  one.  Here  we  will  discuss  some  of  the 
considerations  surrounding  this  notion,  and  attempt  to  establish  rea- 
sonable criteria  for  ultimately  eliciting  a collection  of  expert  judg- 
ments on  the  same  sets  of  questions.  The  basic  considerations  can  be 
divided  into  four  broad  categories:  the  existence  of  expertise;  the 

identification  and  selection  of  a panel  of  experts;  the  formalism  and 
procedural  questions  associated  with  eliciting  responses;  and  the  de- 
sign of  a meaningful  measuring  instrument  (a  survey  questionnaire). 
These  categories  are  discussed  below. 

EXISTENCE  OF  EXPERTISE 

In  many  instances  it  is  difficult  to  argue  that  there  is  such  a 
thing  as  expertise.  For  example,  suppose  it  is  6 months  before  a na- 
tional election  and  the  question  is,  "Who  will  win  the  race?"  No  one 
really  knows , and  the  knowledge  of  people  who  make  a career  out  of 
studying  elections  is  not  significantly  greater,  at  this  time,  than 
that  of  the  average  person.  In  another  context,  suppose  we  are 

12 

One  mechanism  for  eliciting  the  judgments  of  experts  on  fuzzy 
issues  and  then  grouping  them  to  obtain  a consensus  has  been  the 
Delphi  Technique.  The  survey  methods  discussed  below  differ  in  many 
important  respects  from  the  conventional  Delphi  approach,  as  will  be- 
come apparent. 
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in  teres  ted  in  speculating  about  the  "qualities  of  eveiyday  life"  in  the 
year  2000  A.D.  It  is  difficult  to  imagine  that  a few  persons  possess 
a greater  degree  of  knowledge,  intuitive  understanding,  and  ability 
to  predict  such  "qualities"  than  the  rest  of  us,  (In  fact,  if  such 
greater  knowledge  does  exist,  it  is  not  quite  clear  how  it  can  be 
identified — but  that  is  yet  another  kind  of  problem,  and  one  that  is 
considered  below,) 

The  basic  idea  behind  expertise  is  that  there  are  people  who  have 
considerable  knowledge  and  understanding  of  the  mechanisms  underlying 
particular  problems  and  thus  can  do  an  appreciably  better  job  of  fore- 
casting long-term  trends  and  changes  than  the  non-expert  layman.  The 
notion  of  visiting  the  oracle  at  Delphi  to  receive  "expert"  advice  is 
an  old  one.  We  make  the  same  mistake  today  of  thinking  that  for  every 
problem,  there  exists  an  expert  problemsolver. 

There  is  no  doubt,  however,  that  expertise  does  exist  for  some 
problems.  A good  physician,  for  example,  can  do  a better  job  than  a 
layman  in  assessing  the  likelihood  that  a given  individual  will  develop 
cancer;  similarly,  it  is  easier  for  a good  lawyer  to  predict  the  likely 
behavior  of  a judge  or  jury,  in  a given  context,  than  a layman,  and  a 
good  scientist  or  engineer  can  more  accurately  forecast  technological 
change  than  someone  without  technical  training  and  experience. 

In  the  context  of  an  R&D  budget,  once  the  planning  objectives  are 
carefully  defined,  the  first  step  in  an  evaluation  is  to  determine 
whether  or  not  expertise  really  exists.  If  it  does  not,  we  must  seek 
a new  path  for  evaluation;  if  it  does,  we  can  proceed  to  the  next  set 
of  considerations. 

IDENTIFICATION  AND  SELECTION  OF  A PANEL  OF  EXPERTS 

Having  determined  that  expertise  does  exist  for  the  problem  at 
hand,  we  must  now  pick  a panel  of  experts,  A number  of  questions 
arise:  How  do  we  recognize  an  expert?  How  heavily  should  each  opin- 

ion be  weighted?  How  many  experts  constitute  a "good"  panel?-  Should 
all  experts  have  the  same  type  of  expertise?  How  many  panelists  with 
each  type  of  expertise  should  there  be?  Do  the  experts  believe  they 
are  experts  and  how  does  that  affect  their  judgment?  What  are  the 
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common  characteristics  of  experts?  In  selecting  a panel,  should  their 
qualities  be  matched? 

Attributes  of  a "Good"  Panel  of  Experts 

We  believe  that  several  characteristics  are  important  for  panels 
charged  with  assessing  potential  developments  associated  with  techno- 
logical change: 

1.  Diversity:  Panel  members  as  a group  reflect  a wide  spectrum 

of  talents.  Good  panels  are  not  monolithic  in  terms  of  the 
fields  of  expertise  represented.  Rather,  for  every  aspect 
of  the  problem  under  study,  there  is  a panel  member  who  is 
expert  in  that  area.  This  representative  diversity  of  dis- 
ciplines is  necessary  so  that  the  panel  will  not  overlook,  or 
give  perfunctory  treatment  to,  fundamentally  important  elements 
of  the  problem. 

2.  Depth:  Some  panel  members  have  a profound  understanding  of 

the  technical  issues  involved  in  certain  problem  areas.  They 
are  considerably  more  knowledgeable,  in  a scientific  sense, 
than  most  people  in  the  world,  in  their  particular  specialty. 
Thus,  for  every  major  scientific  area  that  is  a component  of 
the  basic  problem,  there  is  at  least  one  panel  expert  who  un- 
derstands that  component  in  depth. 

3.  Breadth : Good  panels  contain  some  members  who  are  "systems 

experts,"  i.e.,  individuals  who  are  accustomed  to  thinking 

on  a broad  level  in  terms  of  the  interactions  of  various  sub- 
systems, of  the  implications  of  new  subsystem  developments  on 
the  economic  feasibility  of  an  entire  system,  and  of  the  po- 
litical, legal,  social,  and  ecological  aspects  of  the  new  de- 
velopment. Panel  members  who  have  a breadth  of  knowledge  are 
better  able  to  predict  the  feasibility  and  likelihood  of  a 
major  technological  breakthrough  than,  say,  a "deeply  knowl- 
edgeable" scientific  expert  who,  in  his  narrower  view,  may 
ignore  the  need  for  developments  in  other  fields  that  will 
make  those  in  his  own  more  meaningful. 
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There  is  no  clearly  defined  "best  mix"  of  experts  for  a given 
panel:  What  fraction  of  the  panel  members  should  be  systems  people 

and  what  fraction  discipline  experts  cannot  be  precisely  stated.  But 
we  can  at  least  establish  lower  bounds.  Once  the  problem  has  been 
broken  down  into  some  well-defined  fields  in  which  expertise  exists, 
we  know  that  there  probably  ought  to  be  at  least  one  expert  from  each 
field,  and  at  least  one  systems  analyst  whose  expertise  has  emerged 
out  of  that  field. 

Identification  of  Experts 

A reasonable  definition  of  expertise  includes  recognition  and  ap- 
probation by  peer  groups:  a person  is  an  expert  in  his  field  if  others 

in  his  field  consider  him  to  be  one.  Some  measures  of  expertise,  by 
this  definition,  are  the  holding  of  office  in  a national  scientific 
organization;  a position  on  the  editorial  board  of  important  technical 
journals  in  the  field;  awards  for  outstanding  scholarship;  honorary 
positions  in  national  societies;  publications  of  non-introductory  books 
(monographs  and  advanced  treatises);  and  awards  of  research  contracts 
from  various  branches  of  the  federal  government.  When  a variety  of 
professionals  in  a field  are  polled  about  whom  they  regard  as  an  ex- 
pert, and  the  same  individuals  are  repeatedly  mentioned,  those  indi- 
viduals must  be  considered  experts. 

Outstanding  systems  analysts  have  typically  been  technical  experts 
at  one  time  and  have  later  switched  to  administration  for  personal  de- 
velopment, As  a result,  their  perspective  has  broadened  and  their 
knowledge  of  related  fields  has  increased,  while  their  awareness  of  de- 
tailed developments  in  their  original  field  has  perhaps  decreased.  As 
they  have  interacted  more  with  people  from  other  disciplines,  however, 
they  have  increasingly  found  instances  where  the  dominating  constraint 
on  a development  has  involved  some  field  other  than  their  own.  After 
coping  with  many  diverse  developmental  efforts,  these  analysts  have 
become  adept  at  anticipating  the  feasibility,  timing,  and  lik_ely  con- 
straints associated  with  any  new  technological  construct.  They  are 
usually  employed  as  some  type  of  manager  (academic  department  chair- 
man, research  director,  R&D  manager  for  a corporation  or  a governmental 
agency,  etc.). 
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Systems  analysts  are  extremely  valuable  on  a panel  not  only  be- 
cause of  their  broad  perspective,  but  also  because  they  tend  to  coun- 
terbalance the  very  conservative  viewpoints  typically  found  among  sci- 
entific experts.  These  latter  individuals,  with  their  deep  knowledge 
of  a subject,  have  been  indoctrinated  throughout  the  years  to  exercise 
extreme  caution  (if  not  suspicion)  about  scientific  breakthroughs  and 
meaningful  technological  advances.  They  are  not  inclined  to  make  in- 
tuitive judgments  about  broad  issues,  and  so  as  a group  they  tend  to 
be  conservative  about  the  feasibility,  timing,  and  costs  of  new  de- 
velopments. Such  a posture  is  "safe."  If  they  are  wrong,  they  won’t 
be  absurdly  wrong  and  thus  subject  to  criticism  and  loss  of  status. 
Systems  analysts,  on  the  other  hand,  tend  to  be  less  conservative. 

They  will,  for  example,  often  substantially  reduce  the  time  estimates 
for  a new  development  given  to  them  by  a scientific  expert  on  their 
staff. 

Selection  of  Panel  Members 

To  minimize  selection  bias,  panelists  should  be  chosen  by  standard 

13 

procedures  developed  in  statistics  and  the  theory  of  psychological 
14 

measurement.  It  is  easy  to  see  how  careless  selection  methods  could 

reflect  institutional  rivalries  and  the  personal  biases  of  those  who 

do  the  selecting.  What  is  needed  to  start  with  is  a listing  of  all 

known  experts  in  each  of  the  fields  required  for  the  analysis,  and  a 

similar  listing  for  the  systems  people.  This  will  establish  popula- 
15 

tions  of  experts.  Then,  after  stratifying  by  field,  simple  random 
samples  can  be  taken  from  each  list  (population).  One  alternative 
might  be  to  stratify  still  further  by  preparing  lists  of  experts  in 
_ 

See,  for  example,  M.  W.  Hansen,  W.  N.  Hurwitz,  and  W.  G.  Madow, 
Sample  Survey  Methods  and  Theory , New  York:  John  Wiley  & Sons,  Inc., 

1953. 

^See,  for  example,  American  Psychological  Association,  Standards 
for  Educational  and  Psychological  Tests  and  Manuals , Washington,  D.C., 
1966. 

■^While  it  is  conceivable  that  it  may  be  desirable  to  stratify 
these  populations  by  levels  of  expertise,  one  person’s  assessment  is 
rarely  appropriate. 
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government  , industry,  and  the  academic  world  for  each  field.  We  could 
then  choose  a simple  random  sample  from  each  of  these  categories  for 
each  field.  Systems  analysts  should  also  be  chosen  for  the  panel  by 
stratified  random  sampling,  using  the  same  procedures.  By  using  random 
number  tables,  in  the  usual  way,  to  choose  a random  sample  from  these 
population  lists,  it  should  be  possible  to  obtain  judgments  that  are 
representative  of  those  of  the  entire  list,  and  to  obtain  the  same  re- 
sults if  the  survey  is  carried  out  on  several  more  occasions  with  simi- 
larly chosen  samples.  Moreover,  the  viewpoints  or  biases  characteris- 
tic of  a certain  class  of  expertise  (e.g.,  originating  in  industry) 
can  be  separately  investigated  and  appropriate  allowances  made. 

Motivation  of  Panel  Members 

A very  important  issue  associated  with  panel  selection  concerns 
the  motivation  of  the  panel  members  to  participate  fully  in  the  study. 
Suppose,  for  example,  we  have  two  experts  in  the  same  field  who,  for 
our  purposes,  are  equivalent  in  expertise  and  we  wish  to  compare  their 
responses  to  a given  question.  If  one  expert  gives  an  of f-the-top-of- 
the-head  response  (a  response  based  upon  a few  seconds  or  minutes  of 
thinking  and  intuiting),  while  the  other  thinks  through  all  the  steps 
necessary  to  reach  the  final  goal  or  development,  evaluating  the  prob- 
lems and  constraints  associated  with  each  stage  and  assessing  condi- 
tional probabilities,  it  seems  reasonable  that  we  should  weight  the 
careful  judgment  more  heavily.  If  the  panelists  are  not  strongly  mo- 
tivated to  cooperate  fully,  to  the  extent  of  providing  careful,  intro- 
spective responses,  the  results  of  the  survey  will  not  be  reliable. 

Of  course  we  can  ask  the  panelist,  on  the  questionnaire,  how  much  time 
he  devoted  to  preparing  his  responses;  and  we  can  word  some  of  the 
technical  questions  so  that  the  panelist  is  required  to  provide  step- 
by-step  responses  as  often  as  possible.  But  these  approaches,  while 
helpful,  don’t  really  come  to  grips  with  the  source  of  the  difficulty, 
namely,  the  panelist’s  lack  of  motivation. 

Various  means  might  be  used  to  induce  experts  to  respond  coopera- 
tively (assuming  they  agree  to  participate  in  the  study  in  the  first 
place).  Possible  motivational  techniques  include: 
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1.  Honoraria:  Participants  could  be  paid  a token  fee.  Such  a 
fee  could  hardly  be  less  than  $50  or  $100.  If  there  were  30 
panel  participants,  this  would  imply  an  honorarium  cost  of 
$1500  to  $3000.  But  a token  fee  is  too  small  to  represent 
any  real  inducement  to  introspection;  it  is  more  likely  to 
induce  some  panelists  to  agree  to  participate  in  the  study 
with  the  intention  of  providing  only  $50  worth  of  effort.  A 
significant  fee  of,  say,  $1000  or  more  could  bring  the  cost 
of  the  study  to  $30,000  or  more,  which  might  be  prohibitive. 

The  larger  fee,  however,  is  much  more  likely  to  induce  moti- 
vation than  a simple  honorarium.  (Clearly,  a $50,000  fee 
would  induce  almost  any  scientist  to  be  very  conscientious; 
but  how  much  less  will  do  the  same  job?) 

2.  Participant  listing:  The  participants  might  be  promised  that 
they  would  be  listed  as  a group  in  the  written  report  of  the 
study  results.  Such  an  approach  is  in  keeping  with  the  no- 
tion that  seeing  onefs  name  in  print  is  a strong  inducement 
to  making  sure  that  the  work  associated  with  his  name  is  sub- 
stantial and  correct.  However,  since  the  resulting  report 
will  not  have  the  status  of  a paper  in  a prestigious  scientific 
journal,  nor  will  the  individual  be  spotlighted  as  an  innovator 
but  will  only  be  listed  as  one  of  a group  whose  opinion  was 
sought,  the  motivating  effect  of  such  a listing  is  not  likely 
to  be  significant. 

3.  Real-time  interactions:  The  procedures  by  which  information 

is  elicited  may  influence  its  quality.  A telephone  interview 
by  a technically  competent  interviewer  can  assess  the  serious- 
ness with  which  the  panelist  is  considering  the  questions  and 
at  the  same  time  probe  to  determine  if  certain  subtleties  of 
the  questions  have  been  comprehended.  However,  the  perspec- 
tive of  the  interviewer  himself  may  unduly  influence  the 
panelist.  An  alternative  is  real-time  response  and  interac- 
tion among  the  panelists  via  a distributed  computer  system 

on  which  individuals  would  probe  for  the  reasons  behind  each 
other1 s assessments  as  well  as  for  characteristics  of  the  group 
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of  responses.  This  procedure  might  tend  to  encourage  group 
self-motivation  without  the  personal  interactions  often  at- 
tributed to  in-person  panel  discussions. 

4.  Research  contract  award  potential:  If  the  request  to  partici- 

pate in  the  study  comes  from  a potential  source  of  research 
contract  funds,  the  inducements  to  many  individuals  to  coop- 
erate in  the  study  may  be  very  great.  This  would  be  particu- 
larly true  if  all  questionnaires  were  name-tagged  and  it  was 
clear  to  every  participant  that  the  survey  monitors  were  keep- 
ing track  of  how  each  individual  was  responding.  Such  a mo- 
tivation may  also  lead  to  biases  in  the  responses,  as  noted 
above. 

PROCEDURAL  ISSUES 

Once  an  appropriate  panel  of  respondents  has  been  selected  we  must 
be  concerned  with  some  procedural  and  administrative  issues.  Should 
the  questions  of  interest  be  asked  in  personal  interviews  with  con- 
tinual interaction  and  feedback  between  interviewer  and  interviewee? 
Should  the  questions  be  asked  by  telephone,  or  by  mail?  (There  are  ad- 
vantages and  disadvantages  to  each  of  these  approaches.)  Should  some 
questions  be  addressed  to  panel  members  unknown  to  one  another  (so  that 
their  judgments  cannot  be  impugned  on  an  authoritarian  basis)?  Should 
some  questions  be  addressed  in  group  discussions,  with  all  panel  members 
freely  interacting  with  one  another,  airing  their  views  openly?  Should 
there  be  one  fixed  set  of  questions  or  should  the  questions  proceed  in 
stages  on  one  basis  or  another?  Should  panelists  be  asked  the  same 
questions  repeatedly,  after  telling  them  the  opinions  of  other  respon- 
dents? 

How  can  we  ensure  that  the  questions  are  valid,  i.e.,  that  they 
are  phrased  in  such  a way  that  they  are  really  providing  answers  to 
the  questions  we  want  answered,  with  minimum  semantical  difficulties 
and  maximum  focus  on  the  true  points  of  interest?  A well-known  method 
for  checking  validity  is  to  use  the  response  results  for  forecasting 
and  to  compare  these  forecasts  with  actual  outcomes.  But  when  we  are 
in  a forecasting  context  in  the  first  place,  this  approach  may  be 
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difficult  to  implement.  A pilot  study  involving  short-term  prediction 
might  prove  helpful. 

The  choice  of  a correct  administrative  approach  to  be  used  in 
eliciting  responses  from  a panel  of  scientific  experts  depends  very 
much  on  the  type  of  question  being  asked  and  on  the  use  to  which  the 
responses  will  ultimately  be  put.  It  seems  reasonable  to  expect  that 
questions  involving  detailed  scientific  knowledge  and  expertise  are 
best  answered  individually,  with  a minimum  of  outside  bias  from  other 
panelists,  from  an  interviewer,  or  from  supervisory  personnel.  Other 
questions,  such  as  those  involving  the  potential  use  of  policy  vari- 
ables that  might  greatly  affect  rates  of  technological  change  and  de- 
velopment, are  probably  best  handled  in  group  discussions.  For  ex- 
ample, if  a panelist  knew  that  his  particular  research,  while  apparently 
not  very  important  in  and  of  itself,  was  in  fact  the  major  limiting 
factor  to  an  extremely  important  development,  and  if  he  also  knew  that 
because  of  its  importance,  the  funding  level  of  support  for  his  research 
might  be  increased  100  percent,  his  judgment  about  the  feasibility  and 
timing  of  some  theoretical  future  development  might  be  drastically  al- 
tered. This  type  of  background  information,  while  difficult  to  supply 
in  a questionnaire  (since  we  can’t  always  anticipate  all  the  implicit 
questions  asked,  and  the  underlying  assumptions  made,  by  a respondent), 
is  quickly  requested  and  supplied  in  a group  discussion. 

The  survey  questionnaires  might  be  administered  by  preceding  their 
mailing  with  letters  and  or  telephone  calls  advising  that  they  are 
going  to  be  mailed,  followed  by  postmailing  telephone  calls'^  to  check 
for  their  receipt  and  to  clarify  any  questions.  Such  a procedure  should 
help  to  ensure  the  validity  of  the  questions  and  minimize  non-response; 
it  should  also  help  to  stimulate  the  respondents’  introspection  and 
cooperation. 

Questions  might  be  asked  once,  or  they  might  be  repeated  in  stages 
in  a controlled  way  (see  Section  IV) . There  are  many  advantages  to 
controlled  feedback.  For  example,  by  requiring  every  panelist  to 

^In  the  basic  mailing,  panelists  could  be  told  that  there  would 
be  a telephone  followup. 
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pro  vide  some  discussion  (say,  a paragraph  of  prose)  about  why  he  be- 
lieves in  his  first-round  position,  and  then  later,  why  he  is  either 
adhering  to  this  position  or  changing  from  it  (after  having  been  sup- 
plied with  a complete  set  of  first-round  responses  from  all  respondents) 
we  are  forcing  all  panelists  to  think  through  their  judgments  very 
carefully,  vis-a-vis  all  other  panelists,  and  we  are  zeroing  in  on  the 
best  rationale  for  the  group  judgments.  Certainly  no  type  of  unanimous 

(or  even  consensus)  judgment  should  be  required  for  this  type  of  anal- 

. 17 

ysis . 

Questions  involving  a paragraph  of  prose  or  an  enumeration  of  rea- 
sons are  useful  for  collecting  ideas  about  how  to  regard  an  issue.  We 
believe  questions  of  this  type  are  probably  the  most  reasonable  for 
the  first  stage  of  such  a study.  Later  stages  might  involve  more 
precisely  focused  questions  to  the  same  panel. 

INSTRUMENT  DESIGN 

The  design  of  a suitable  instrument  for  measuring  the  judgments 
of  scientific  experts  about  the  feasibility  of  some  technological  de- 
velopments depends,  of  course,  to  a great  extent  on  the  specific  de- 
velopments we  are  interested  in,  the  breadth  and  depth  of  the  type  of 
information  sought,  and  the  degree  of  detail  required.  However,  at 
this  preliminary  stage,  prior  to  actual  design,  we  can  at  least  estab- 
lish some  guidelines  and  general  considerations  that  should  prove  useful 

18 

Questionnaire  design  is  an  art  about  which  much  has  been  written. 
There  have  been  many  guiding  principles  laid  down  on  the  basis  of  past 
experience  that  we  cannot  afford  to  ignore.  They  include  the  importance 
of  giving  the  instrument  a preliminary  trial  run  in  a pilot  program, 
the  care  that  must  be  exercised  in  wording  questions,  and  the  close 

"^Controlled  feedback  is  also  used  in  the  Delphi  approach  (see 
Section  IV).  However,  there  it  is  typically  used  by  asking  panelists 
to  justify  their  positions  only  if  they  are  outside  the  interquartile 
range  of  the  distribution  of  responses.  Such  an  approach,  of-  course, 
encourages  agreement  even  when  it  may  not  be  appropriate. 

18 

See,  for  example,  A.  N.  Oppenheim,  Questionnaire  Design  and  At- 
titude Measurements , New  York:  Basic  Books,  Inc.,  1966. 
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attention  that  must  be  paid  to  the  design  of  scales  for  recording  judg- 
ments with  quantitative  content.  Without  commenting  further  on  these 
important,  but  fairly  standard,  problems  associated  with  all  question- 
naire designs,  we  will  move  on  to  discussion  of  some  of  the  problems 
peculiar  to  our  context. 

Types  of  Questions 

It  seems  appropriate,  in  our  problem,  that  the  survey  questionnaire 
include  at  least  three  distinct  categories  of  questions: 

1.  Questions  dealing  with  the  backgrounds  of  the  individual 
panelists. 

2.  Questions  dealing  with  the  nature,  format,  and  administration 
of  the  questionnaire. 

3.  Questions  dealing  with  the  scientific  content  motivating 
the  entire  study. 

Questions  in  the  first  category  are  aimed  at  determining  the 
panelist’s  level  of  expertise,  his  biases,  the  nature  of  his  background 
(e.g.,  is  he  a systems  manager?)  and  in  general  his  qualifications  for 
being  on  the  panel.  There  might  also  be  some  questions  directed  at  as- 
sessing his  scientific  conservatism.  Asking  for  his  judgment  on  a key 
scientific  question,  for  which  there  already  exists  a known  spectrum  of 
viewpoints,  might  be  a useful  way  of  assessing  his  conservatism  rela- 
tive to  that  of  the  other  panelists.  The  biases  of  a panelist  might 
be  revealed  by  asking  for  a listing  of  his  best  publications.  Answers 
to  this  group  of  questions  should  provide  some  useful  insights  for  un- 
derstanding and  interpreting  the  quality  and  perspective  of  each  in- 
dividual’s responses. 

Questions  in  the  second  category  attempt  to  assess  (a)  the  care 
that  was  exercised  in  completing  the  questionnaire,  (b)  whether  or 
not  the  questions  were  clear,  (c)  how  the  question  format  might  be  im- 
proved in  the  next  round,  and  (d)  whether  or  not  the  panelist  feels 
he  was  too  constrained  in  his  responses  by  the  way  in  which  the  ques- 
tionnaire was  administered. 
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Questionnaire  Goals 

The  goal  of  the  survey  questionnaire  is  to  assist  the  manager  in 
allocating  his  R&D  budget.  It  is  therefore  desirable  to  condition  the 
questions,  whenever  possible,  on  policy  issues.  For  example,  in  asking 
a panelist  to  assess  the  feasibility  of  some  new  development,  he  might 
be  asked  for  three  separate  assessments;  one  assuming  a "low”  funding 
level  for  the  necessary  research,  a second  assuming  a "medium"  funding 
level,  and  a third  assuming  a "high"  funding  level  (of  course,  these 
terms  need  to  be  suitably  defined  either  as  part  of  the  questionnaire, 
or  as  a part  of  the  required  response).  It  might  be  worthwhile  to  ask 
if  a given  development  is  likely  within,  say,  2 years,  5 years,  10  years, 
or  perhaps  never.  It  would  also  be  useful  to  ask  what  other  areas  of 
research  need  to  be  "stimulated"  because  of  their  likely  complementary 
payoffs.  These  other  areas  might  only  relate  to  peripheral  aspects  of 
the  panelist’s  own  work,  or  broaden  its  applicability. 

Questionnaire  Format 

An  important  problem  in  this  type  of  study  is  how  to  phrase  the 
questions  so  that  they  cause  the  panelist  to  reflect  thoughtfully 
about  the  problems  and  their  constraints  and  limitations. 

One  set  of  questions  might  permit  the  panelist  infinite  latitude 
by  asking  him  to  enumerate  all  the  steps  that  would  be  required  to 
attain  a given  technological  development.  Another  set,  which  might 
constrain  him  somewhat  more,  would  provide  him  with  some  of  the  basic 
steps  required  to  attain  a given  development,  but  would  ask  him  to  add 
or  delete  steps,  as  appropriate,  and  to  assess  conditional  probabili- 
ties  of  being  able  to  proceed  down  the  chain  of  steps  at  each  stage. 

The  end  result  would  be  an  "achievement  tree"  with  many  nodal  points 
and  assessments  of  the  conditional  probabilities  of  moving  between 
any  two  nodal  points.  The  degree  of  detail  could  be  refined  in  suc- 
cessive stages  of  the  questioning,  as  could  the  subjective  probability 
assessments.  Moreover,  successive  stages  of  questioning  could  easily 
lead  to  the  emergence  of  completely  new  paths  of  development. 
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IV.  QUANTIFICATION  AND  ANALYSIS  OF  GROUP  JUDGMENT  DATA 

Research  on  quantitative  assessment  of  technological  advancement 

by  the  use  of  expert  judgments  appears  to  have  had  its  formal  genesis 

19  20 

in  studies  carried  out  by  Marshall  and  Meckling,  Klein,  and 
21 

Summers,  who  each  made  use  of  a quantity  "A, 11  the  degree  of  techno- 
logical advancement  sought  in  a program.  To  estimate  A,  a sample  sur- 
vey was  taken  using  four  "experienced  Rand  engineers"  as  sample  ele- 
ments. The  four  subjects  were  asked  to  rate  subjectively  the  magnitude 
of  the  improvement  in  the  state  of  the  art  required  for  each  of  22 
aircraft  and  missile  development  programs.  Their  ratings  were  to  be 
placed  on  a numerical  scale  ranging  from  1 to  4.  Group  judgments  were 
assessed  by  summing  the  ratings  of  the  four  subjects.  After  all  the 
ratings  were  obtained,  each  program  was  categorized  as  small,  medium, 
or  large  in  the  Marshall  and  Meckling  and  Klein  studies.  In  the 
Summers  study,  the  A variable  (henceforth  called  the  A-f actor)  and 
others  were  related  to  cost  factors  (ratios  of  actual  to  estimated 
costs)  of  a program  by  means  of  standard  regression  techniques. 

Subjective  assessments  of  the  A-factor  were  attempted  in  two  sub- 
sequent surveys.  One  was  reported  on  by  Harman  and  Henrichsen  in 
22 

1970.  The  sample  subjects  were  asked  to  assess  the  A-factor  for  air- 
craft and  missile  systems  on  a scale  of  0 to  20.  As  in  the  earlier 

19 

A.  W.  Marshall  and  W.  H.  Meckling,  Predictability  of  the  Costs , 

Time , and  Success  of  Development , The  Rand  Corporation,  P-1821,  Decem- 
ber 1959,  p.  17;  also  published  in  R.  R.  Nelson  (ed.).  The  Rate  and 
Direction  of  Inventive  Activity:  Economic  and  Social  Factors,  Prince- 

ton, Princeton  University  Press,  1962. 

20 

B.  H.  Klein,  "The  Decision  Making  Problem  in  Development,"  in 
R.  R.  Nelson,  op.  cit. 

21 

R.  Summers,  Cost  Estimates  as  Predictors  of  Actual  Weapon  Costs: 
A Study  of  Major  Hardware  Articles , The  Rand  Corporation,  RM-3061-PR 
(Abridged),  March  1965. 

22 

A.  J.  Harman,  assisted  by  S.  Henrichsen,  A Methodology  for  Cost 
Factor  Comparison  and  Prediction , The  Rand  Corporation,  RM-6269-ARPA, 
August  1970. 
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survey,  the  subjects  were  experienced  Rand  engineers,  and  the  sample 
size  for  any  given  system  ranged  from  two  to  four  persons* 

The  last  of  the  three  surveys  was  made  in  1970  in  connection  with 

23 

the  aircraft  turbine  engine*  In  this  survey,  A-f actors  were  assessed 
(among  other  things)  for  aircraft  turbine  engines  on  a scale  of  1 to 
20*  The  eleven  subjects  were  all  employees  of  the  General  Electric 
Company  (some  of  the  systems  evaluated  were  manufactured  by  G.E.  and 
some  were  not).  Response  was  small  and  not  all  subjects  made  assess- 
ments for  all  systems. 

Although  the  three  surveys  undertaken  to  assess  A-f actors  have 
contributed  considerably  to  our  understanding  of  the  problem  of  how 
to  measure  technological  advancement,  they  still  leave  much  to  be  de- 
sired. For  example,  samples  have  been  so  small  that  it  is  difficult 
to  make  meaningful  statistical  statements  about  the  results.  Also 
there  are  problems  associated  with  asking  individuals  (no  matter  how 
expert)  to  compare  objects  having  many  characteristics  or  attributes 
on  a single  numerical  scale.  Not  only  will  individual  perceptions  of 
reality  tend  to  be  different,  but  so  will  individual  weights  assigned 
to  each  attribute  of  an  object  being  studied.  Finally,  individuals 
differ  in  their  ability  to  quantify  their  judgments;  even  though  they 
may  all  view  an  object  in  the  same  way,  they  may  very  well  differ  in 
their  quantitative  description  of  it. 

This  section  discusses  three  separate  methods  of  solving  the  prob- 
lem of  assessing  technological  advancement  by  quantifying  sets  of  judg- 
ments: (1)  a type  of  multidimensional  scaling  called  "individual- 

differences  scaling,"  (2)  subjective  probability  assessment  techniques, 
and  (3)  multivariate  regression  with  categorical  dependent  variables. 
These  methods  have  different  properties,  and  each  is  useful  for  bring- 
ing out  different  types  of  relationships.  Which  method  is  most  appro- 
priate depends  very  much  on  the  characteristics  of  the  specific  problem, 
the  objectives  of  the  study,  and  the  form  of  the  information  available. 
For  some  problems,  one  particular  method  will  be  indicated,  whereas 

23 

A.  J.  Alexander  and  J.  R.  Nelson,  Measuring  Technological  Change: 
Aircraft  Turbine  Engines,  The  Rand  Corporation,  R-1017-ARPA/PR,  June 
1972. 
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for  another , each  of  the  methods  may  be  appropriate  at  a different 
stage  of  the  study. 

The  method  employing  multivariate  regression  with  categorical  de- 
pendent variables  is  new  and  was  developed  specifically  for  this  ap- 
plication. The  mathematical  description  of  the  regression  model  in 
this  section  is  only  intended  to  be  illustrative;  a more  extensive  and 
general  presentation  appears  in  the  appendix. 

The  individual-dif ferences-scaling  method  is  useful  for  placing 
the  various  systems  to  be  compared  in  a multidimensional  coordinate 
frame  of  reference  so  that  the  positions  may  be  compared  on  each  axis. 
Moreover,  the  characteristics  of  the  assessors  may  be  related  to  the 
underlying  dimensions  of  the  system  attributes  so  that  certain  groups 
of  assessors  may  be  found  to  rate  Project  A more  advanced  than  Project 
B on  the  first  two  dimensions,  but  not  on  other  dimensions. 

Subjective  probability  methods  (the  Bayesian  approach)  are  useful 
for  providing  a formal  mechanism  for  combining  objective  data  with 
prior  information  (before  observing  the  objective  data);  of  course, 
in  some  cases  there  are  no  objective  data  available  and  we  must  rely 
solely  on  prior  information.  For  example,  cost  relationships  from 
other  types  of  projects  may  be  used  for  present  projects  even  though 
the  projects  differ  from  one  another.  We  may  have  cost  information 
for  one  type  of  project,  but  not  for  another.  Prior  information  may 
be  introduced  in  the  form  of  A-f actors  or  in  terms  of  probabilities. 
These  approaches  are  compared  later  in  this  section. 

In  the  multivariate  regression  method,  a relationship  is  estab- 
lished between  the  probabilities  that  certain  events  will  occur  (or 
that  certain  propositions  are  true)  and  other  attribute  or  explanatory 
variables  related  to  these  probabilities.  For  example,  it  may  be  of 
interest  to  relate  the  probability  that  a certain  policy  for  funding 
R&D  projects  will  result  in  a defined  product,  within  a preassigned 
time,  to  explanatory  variables  such  as  funding  rate,  length  of  time 
for  which  the  project  is  funded,  type  of  project  supervision  (degree 
of  closeness),  and  other  variables.  This  sort  of  relationship  can  be 
established  on  the  basis  of  a sample  of  previous  data.  Then,  predic- 
tions may  be  suggested  by  the  model. 
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The  three  methods,  applicable  to  different  types  of  problems,  are 
discussed  below. 

MULTIDIMENSIONAL  SCALING  OF  INDIVIDUAL  DIFFERENCES 

A recently  developed  and  very  powerful  method  of  integrating  col- 
lections of  comparative  judgments  of  individuals  to  form  a composite 
group  judgment,  scaled  on  each  of  several  dimensions,  is  called 

r\  t 

"individual-differences  scaling.11  The  basic  idea,  applied  in  one 
context,  is  that  each  subject  compares  N projects  regarding  their  rela- 
tive  degrees  of  technological  advancement.  Thus,  each  individual 
renders  N(N  - l)/2  judgments  of  the  form:  project  S^  is  more  techno- 
logically advanced  than  project  S_.,  for  all  i,  j = 1,  2,...,N.  In 
another  context,  each  individual  may  order  proposed  R&D  projects  accord- 
ing to  their  probability  of  feasible  development  to  a given  stage  by 
a given  date.  Next  it  is  assumed  that  p dimensions  are  sufficient  to 
represent  the  structure  underlying  the  project  differences.  The  ordinal 
judgments  of  a given  individual  can  now  be  represented  as  ranks,  or 

they  can  be  converted  into  "distances"  by  one  of  several  standard  pro- 

26 

cedures,  such  as  by  the  "law  of  comparative  judgments."  Distances 
can  be  represented  as  weighted  distances  in  Euclidian  space,  and  the 
weights  on  each  axis  and  the  coordinates  of  each  point  can  be  estimated 
by  the  data.  Thus,  if  the  distance  between  project  i and  project  j, 
as  perceived  by  subject  k,  is 

24 

J.  D.  Carroll  and  J.  J.  Chang,  "Analysis  of  Individual  Differ- 
ences in  Multidimensional  Scaling  via  an  N-way  Generalization  of 
1 Eckart-Young1  Decomposition,"  Psyahometrika , Vol.  35,  No.  3,  September 
1970,  pp.  283-319. 

25 

The  definition  of  technological  advancement  to  be  used  simulta- 
neously for  past,  current,  and  future  projects  has  not  yet  been  made 
precise. 

26 

See,  for  example,  W.  S.  Torgerson,  Theory  and  Methods  of  Scaling, 
New  York:  John  Wiley  & Sons,  Inc.,  1958;  and  R.  D.  Bock  and  L.  V.  Jones, 

The  Measurement  and  Prediction  of  Judgment  and  Choice , San  Francisco: 
Holden  Day,  1968. 
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i,  j = 1, . . . ,N,  and  k = l,...,n,  if  there  are  n subjects  who  render 
complete  sets  of  judgments.  The  Wj^s  an<^  the  Xjt?s  are  estimated 
from  the  data.  The  result  is  a configuration  of  points  in  p-space 
representing  the  relative  positions  of  the  N systems  as  viewed  by  the 
n subjects,  in  total.  The  coordinates  are  the  scale  values  (on  a ratio 
scale)  of  each  of  the  projects  in  each  dimension.  Thus,  a collection 
of  pairwise  ordinal  rankings  for  each  of  n individuals  would  not  only 
yield  a set  of  combined  judgments  of  ratio-scale  numerical  values  in 
p-dimensions  for  each  project,  but  also  a set  of  weights  for  each  per- 
son in  the  sample,  representing  the  importance  that  person  places  on 
each  axis  for  each  project. 

The  response  of  each  subject  may  be  thought  of  as  being  composed 
of  an  overall  common  mean  response,  plus  a response  effect  reflecting 
his  particular  degree  of  expertise,  plus  a white-noise  error  term  ac- 
counting for  individual  variations  in  ability  to  express  one’s  state 
of  knowledge  (thus,  two  individuals  with  precisely  the  same  state  of 
knowledge  would  still  differ  in  their  responses  because  of  their  error 
terms).  One  implication  of  this  type  of  reasoning  is  that  since  ex- 
perts should  produce  responses  that  are  closer  to  being  correct  than 
non-experts,  great  care  should  be  exercised  in  selecting  the  subjects. 
Once  a panel  of  potential  subjects  is  screened  for  expertise,  choosing 
a sample  from  this  panel,  which  should  be  as  large  as  possible  con- 
sistent with  cost  and  practicality  considerations,  will  result  in  a 
small  "individual-difference  error." 

The  results  of  an  individual-dif ferences-scaling  type  of  analysis 
might  be  used  in  several  ways.  One  way  would  be  to  establish  a func- 
tional relationship  (by  regression  methods)  between  cost  and  the  vari- 
ous dimensions  of  the  projects.  Then,  the  cost  of  a new  project  could 
be  predicted  by  interpolation  (extrapolation).  Another  use  of  the 
results  might  include  changing  the  objectives  sought  in  the  new  project 
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if  it  is  found  that  coordinate  values  along  a certain  axis  are  unrea- 
sonable or  intolerable.  Finally,  after  studying  the  sets  of  estimated 
weighting  factors  obtained  for  each  expert,  a decision  might  be  made 
to  query  certain  individuals  reqarding  their  weighting  of  a particular 
axis.  This  procedure  might  reveal  important  circumstances  related  to 
the  R&D  projects  that  were  overlooked  (or  ignored)  by  other  subjects. 
Perhaps  subjects  should  be  informed  of  the  weights  placed  on  each  axis 
by  others  and  then  the  entire  process  repeated. 


SUBJECTIVE  PROBABILITY  METHODS 


A-Factors  versus  Probabilities 

Technological  advancement  might  be  measured  in  terms  of  A-f actors, 
as  described  above,  or  it  might  be  measured  in  terms  of  probabilities 
of  some  proposition.  For  example,  if  we  speak  of  E^  as  the  proposition 
that  project  S^  will  have,  upon  completion,  A-factor  A^,  i - 1,...,N, 
then  p^  = P{E_^}  should  be  a monotonic  function  of  the  A-factor.  That 
is,  knowing  p^  is  equivalent  to  knowing  the  corresponding  A-factor, 
and  conversely.  The  real  implication  of  this  equivalence  is  that  if 
the  cost  of  a project,  for  example,  is  to  reflect  the  degree  of  its 
technological  sophistication,  either  A-f actors  or  p^fs  could  be  used 
as  independent  (exogenous)  variables  In  the  cost  equations.  It  is  not 
clear  at  this  time  which  of  the  two  is  a better  measure,  in  the  sense 
that  it  can  be  better  estimated  and  can  therefore  be  used  to  generate 
better  cost  predictions.  This  point  will  be  considered  further  below. 

The  relationship  between  the  p/s  and  the  A-f  actors  is  monotonic 
but  not  unique:  there  are  a large  number  of  potentially  useful  and 

convenient  monotonic  functions.  For  example,  a linear  relationship 
is  provided  by 

A.  - a 


where  p^  denotes  the  probability  that  will  have  A-factor  A^,  and 
A^  is  scaled  on  the  interval  [a,b],  where  a denotes  the  minimum  degree 
of  advancement  and  b denotes  the  maximum.  This  relationship  is  de- 
picted in  Fig.  2. 
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Fig*  2 — Linear  correspondence  between  A-factors 
and  probabilities 


Another  potentially  useful  functional  relationship  is  the  logistic 
correspondence,"^  given  by 
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where  0 < 3 < 1, 


-1  1 1 
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a < A < b,  and  a < b.  In  this  relationship,  p^  is  still  a monotonically 
increasing  function  of  A^,  but  the  function  is  convex  or  concave  de- 
pending on  whether  a < A.  < a - log  3,  or  a - log  3 < A < b,  respec- 
tively.  That  is,  there  is  a point  of  inflection  at  A,  = a - log  3. 

The  functional  relationship  is  sketched  in  Fig.  3.  The  appropriate 
value  of  3 might  be  selected  with  the  use  of  regression  techniques, 
after  both  the  A-factor  and  the  probability  information  are  elicited. 

Many  factors  enter  into  the  problem  of  deciding  whether  probabili- 
ties or  A-factors  are  better  measures  of  technological  advancement  (in 
the  estimation  sense  described  above) . Some  of  them  are 


A multidimensional  version  of  this  form  of  correspondence  is 
used  explicitly,  in  a different  context,  in  "Categorical-Dependent- 
Variable  Multivariate  Regression,"  below. 
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Fig • 3 — Logistic  correspondence  between  A-factors 
and  probabilities 

1.  One  may  be  more  easily  assessable  than  the  other  because  it 
may  be  easier  to  get  subjects  to  render  judgments  of  one 
rather  than  the  other. 

2.  One  may  be  more  precisely  assessable  than  the  other  in  terms 
of  repeatability  of  such  assessment  using  different  samples 
of  "equally  competent"  experts;  i.e.,  it  may  be  subject  to 
smaller  variance. 

3.  One  may  be  a more  accurate  measure  than  the  other  in  terms 
of  how  close  to  the  "truth"  the  measure  can  be  expected  to 
come  (on  the  average)  for  a given  sample  size  of  independent 
and  identically  distributed  observations . 

4.  One  measure  may  be  more  improvable  than  the  other,  so  that  it 
may  be  possible,  by  feeding  back  information  to  experts  over 
time,  to  teach  them  to  be  better  assessors  of  one  measure 
than  the  other. 

5.  Individuals  may  vary  less  in  their  ability  to  quantify  their 
judgments  about  linearly  scaled  variables.  There  is  also  the 
difficulty  that  some  subjects  will  have  a better  grasp  of  the 
meaning  and  concept  underlying  a probability  than  others. 

The  ability  of  such  people  to  make  probability  assessments  of 
their  judgments  is  keener  than  that  of  others,  who  might 
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possess  the  same  substantive  information  regarding  a given 
proposition  but  who  are  unable  to  quantify  it  as  well. 


Considerable  research  effort  has  been  devoted  to  finding  methods 

for  getting  subjects  to  assess  scores  for  a variable,  a difficult  prob- 

28 

lem  that  has  consumed  much  time  among  psychologists  and  statisticians. 

The  problem  of  assessing  subjective  probabilities  has  also  received 

2 Q 

considerable  attention  not  only  in  psychological  and  economic  contexts, 

30 

but  also  in  medical  contexts.  However,  whether  probabilities  or  A- 
factors  should  be  assessed  remains  an  open  question. 


See,  for  example,  Torgerson,  op.  cit.;  Bock  and  Jones,  op.  cit.; 
J.  P.  Guilford,  Psychometric  Methods , 2d  ed..  New  York:  McGraw-Hill, 

1954;  and  C.  H.  Coombs,  A Theory  of  Data , New  York:  John  Wiley  & Sons, 

Inc. , 1964 . 
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A brief  bibliography  would  include  W.  Edwards,  "The  Theory  of 
Decision  Making,"  Psychology  Bulletin , Vol.  51,  No.  4,  1954,  pp.  380- 
417;  W.  Edwards,  "The  Reliability  of  Probability  Preferences,"  American 
Journal  of  Psychology , Vol.  67,  1954,  pp.  68-95;  H.  E.  Kyburg,  Jr., 
and  H.  E.  Smokier,  Studies  in  Subjective  Probability , New  York:  John 

Wiley  & Sons,  Inc.,  1964;  M.  G.  Preston  and  P.  Baratta,  "An  Experi- 
mental Study  of  the  Auction-value  of  an  Uncertain  Outcome,"  American 
Journal  of  Psychology , Vol.  61,  1948,  pp.  183-193;  F.  Mosteller  and 
P.  Nogee,  "An  Experimental  Measurement  of  Utility,"  Journal  of  Political 
Economy , Vol.  59,  No.  5,  1951,  pp.  371-404;  R.  L.  Winkler,  "Probabilis- 
tic Prediction:  Some  Experimental  Results,"  Journal  of  the  American 

Statistical  Association , Vol.  66,  1971,  pp.  675-685;  R.  L.  Winkler, 

"The  Quantification  of  Judgment:  Some  Methodological  Suggestions," 

Journal  of  the  American  Statistical  Association , Vol.  62,  1967,  pp. 
1109-1120;  R.  L.  Winkler  and  A.  H.  Murphy,  "’Good1  Probability  Asses- 
sors," Journal  of  Applied  Meteorology , Vol.  7,  1968,  pp.  751-758;  H.  J. 
Einhom,  "Expert  Measurement  and  Mechanical  Combination,"  Organizational 
Behavior  and  Human  Performance , March  1972;  L.  J.  Savage,  "Elicitation 
of  Personal  Probabilities  and  Expectations,"  Journal  of  the  American 
Statistical  Association , Vol.  66,  1971,  pp.  783-801.  For  further  refer- 
ences, see  especially  R.  L.  Winkler,  "Probabilistic  Prediction:  Some 

Experimental  Results,"  op.  cit.,  and  L.  J.  Savage,  op.  cit. 

30 

See,  for  example,  P.  E.  Meehl,  Clinical  vs.  Statistical  Predic- 
tion9  Minneapolis:  University  of  Minnesota  Press,  1954,  and  L.  W. 

Coppleson,  R.  M.  Factor,  S.  B.  Strum,  P.  W.  Graff,  and  H.  Rappaport, 
"Observer  Disagreement  in  the  Classification  and  Histology  of, Hodgkin's 
Disease,"  Journal  of  the  national  Cancer  Institute , Vol.  45,  No.  4, 
October  1970. 
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Bayesian  Approach 

The  Bayesian  approach  to  statistical  inference  and  decisionmaking 

involves  the  assessment  of  prior  distributions  on  the  underlying  pa- 

31 

xameters  of  the  model.  One  of  the  most  important  practical  problems 
associated  with  the  specific  application  of  Bayesian  analysis  is  that 
of  how  best  to  assess  the  prior  distributions.  But  subjective  proba- 
bility assessors  are  very  often  "incoherent11  in  that  their  probability 
assessments  for  various  related  events  are  not  consistent.  For  example, 
for  some  integer  random  variable  E,  an  assessor  might  assert  that  as 
far  as  he  is  concerned,  a priori,  P {E  > 0}  = 0.60  and  P {E  > 10}  = 0.70. 
One  implication  of  these  two  assertions  is  that  P {1  < E ^ 10}  = -0.10, 
an  absurd  result. 

It  is  expected  that  computers  will  be  able  to  assist  in  the  assess- 
ment problem.  That  is,  routines  could  be  developed  to  keep  track  of 
all  previous  assessments  a subject  has  made  about  related  propositions. 
The  computer  could  ask  the  subject  a sequence  of  questions  designed  to 
lead  to  consistent  assessments  of  various  propositions.  If  the  re- 
sponses are  inconsistent,  the  computer  would  indicate  this  and  request 
the  subject  to  be  more  introspective  and  rethink  his  collective  re- 
sponses. Complete  prior  distributions  could  be  assessed  in  this  way. 

This  type  of  computer-assisted  assessment  technique  is  a very  realistic, 

32 

potentially  available  development. 


31  ... 

See,  for  example,  D.  V.  Lindley,  Introduction  to  Probability 

and  Statistics , Vols.  I and  II,  Cambridge:  The  University  Press, 

1965,  for  a discussion  of  Bayesian  inference  in  univariate  analysis, 
and  S.  J.  Press,  Applied  Multivariate  Analysis , New  York:  Holt, 

Rinehart  and  Winston,  Inc. , 1972,  for  its  application  in  multivariate 
analysis.  Some  earlier  work  on  the  assessment  of  subjective  probabili- 
ties, in  the  context  of  weapons  development  and  the  relation  of  sub- 
system properties  to  total  system  performance,  may  be  found  in  F.  S. 
Timson,  Measurement  of  Technical  Performance  in  Weapon  System  Develop- 
ment Programs : Subjective  Probability  Approach , The  Rand  Corporation, 

RM-5207-ARPA,  December  1968. 
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Some  computer-assisted  assessment  techniques  are  already  being 
used  by  M.  Novick  at  the  American  College  Testing  Program,  Iowa  City, 
Iowa. 
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Controlled  Feedback  Methods 

Technological  advance  might  be  assessed  by  some  controlled  feed- 
back method  (the  Delphi  method  is  one  such  approach)  that  seeks  to 
obtain  a consensus  of  a group  of  experts  by  feeding  individual  opinions 
back  to  the  group  so  that  they  can  revise  their  assessments.  Each  ex- 
pert is  typically  subjected  to  a series  of  questionnaires.  The  summary 
statistics  of  the  responses  are  fed  back  to  the  group,  leading  to  a 
new  round  of  revised  responses.  The  feedback  process  is  repeated  sev- 
eral times.  It  is  hoped  that  consensus  will  be  achieved  after  several 
rounds.  The  experts  are  generally  unknown  to  one  another,  and  their 
opinions  are  often  solicited  by  mail. 

The  Delphi  method  was  originally  devised  in  1951  to  apply  expert 
opinion  to  the  selection,  from  the  viewpoint  of  a Soviet  strategic 
planner,  of  an  optimal  U.S.  industrial  target  system  and  to  the  esti- 
mation of  the  number  of  A-bombs  required  to  reduce  the  munitions  output 

33 

by  a prescribed  amount.  Since  then  the  original  Delphi  techniques 

and  many  variations  of  them  have  been  applied  (and  misapplied)  to  a 

diverse  collection  of  problems  ranging  from  technological  forecasting 

for  advance  planning  for  corporations,  to  studying  national  goals  for 

the  United  States  and  for  various  foreign  countries.  Some  of  these 

applications  are  not  ones  for  which  expertise  obviously  exists  (see 

Section  III).  After  study  and  application  of  the  techniques  over  the 

34 

years,  a collection  of  four  summarizing  reports  appeared  at  Rand. 


See  N.  Dalkey  and  0.  Helmer,  The  Use  of  Experts  for  the  Estima- 
tion of  Bombing  Requirements:  A Project  Delphi  Experiment , The  Rand 

Corporation,  RM-727,  November  1951,  and  N.  Dalkey  and  0.  Helmer,  An 
Experimental  Application  of  the  Delphi  Method  to  the  Use  of  Experts , 

The  Rand  Corporation,  RM-727  (Abridged),  July  1962. 

Q / 

N.  C.  Dalkey,  The  Delphi  Method:  An  Experimental  Study  of  Group 

Opinion , The  Rand  Corporation,  BM-5888-PR,  June  1969;  B.  Brown,  S.  W. 
Cochran,  and  N.  C.  Dalkey,  The  Delphi  Method  II:  Structvcre  of  Experi- 

ments, The  Rand  Corporation,  RM-5957-PR,  June  1969;  N.  C.  Dalkey, 

B.  Brown,  and  S.  W.  Cochran,  The  Delphi  Method 3 III:  Use  of  Self- 

Ratings  To  Improve  Group  Estimates , The  Rand  Corporation,  RM-6115-PR, 
November  1969;  N.  C.  Dalkey,  B.  Brown,  and  S.  W.  Cochran,  The  Delphi 
Method y IV:  Effect  of  Percentile  Feedback  and  Feed-in  of  Relevant 

Facts , The  Rand  Corporation,  RM-6118-PR,  March  1970. 
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Other  Rand  studies  have  reported  on  more  recent  experimental  results. 

A critique  of  Delphi  that  focused  on  the  misuse  of  controlled  feedback 
procedures  has  been  completed  recently. 

The  idea  behind  Delphi  and  other  controlled  feedback  techniques 
is  that  if  you  want  the  best  guess  about  a "fuzzy"  question — one  that 
is  extremely  difficult  to  answer  even  for  the  most  informed  people — ask 
an  expert.  Moreover,  since  for  certain  problems  many  expert  heads  are 
better  than  one,  ask  many  experts  and  combine  the  conclusions,  weight- 
ing them  by  the  degree  of  expertise  in  the  subject. 

The  details  of  how  to  implement  this  type  of  philosophy  have  varied 
from  one  application  to  another,  but  the  basic  idea  remains  the  same. 

In  some  applications,  the  median  response  is  fed  back  to  the  subjects 
for  comparison  with  their  own  responses,  and  then  a second-round  re- 
sponse is  sought.  In  other  applications,  upper  and  lower  quartiles 
of  the  responses,  as  well  as  the  median,  are  fed  back.  In  some  appli- 
cations, subjects  whose  responses  fall  outside  the  upper  and  lower 
quartiles  are  requested  to  explain  why  they  are  outliers;  otherwise, 
they  are  required  to  change  their  positions.  In  still  other  applica- 
tions, subjects  are  required  to  extrapolate  the  future  from  earlier 
data,  or  they  are  required  to  provide  paragraphs  of  prose  describing 
their  feelings  or  beliefs  on  an  issue.  It  is  the  last-mentioned  ap- 
proach that  we  believe  to  be  most  appropriate,  for  reasons  summarized 
in  Section  III. 

Various  theoretical  and  practical  questions  need  to  be  raised  and 
answered  concerning  the  analysis  of  data  collected  in  this  way  before 
we  can  seriously  consider  applying  any  controlled  feedback  technique 
to  the  problem  of  assessing  technological  advancement. 
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N.  C.  Dalkey  and  D.  L.  Rourke,  Experimental  Assessment  of  Delphi 
Procedures  with  Group  Value  Judgments , The  Rand  Corporation,  R-612-ARPA, 
February  1971,  and  N.  C.  Dalkey  and  B.  Brown,  Comparison  of  Group  Judg- 
ment Techniques  with  Short-Rccnd  Predictions  and  Almanac  Questions , The 

Rand  Corporation,  R-678-ARPA,  May  1971. 
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H.  Sackman,  Delphi  Assessment:  Expert  Opinion , Forecasting , 

and  Group  Process , The  Rand  Corporation,  R-1283-PR,  April  1974.  See 
also  M.  W.  Hansen,  W.  N.  Hurwitz,  and  W.  G.  Madow,  Sample  Survey 
Methods  and  Theory , New  York:  John  Wiley  & Sons,  1953;  American  Psy- 

chological Association,  Standards  for  Educational  and  Psychological 
Tests  and  Manuals,  Washington,  D.C.,  1966. 
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For  example,  the  distribution  of  responses  in  a Delphi  study  has 

37 

been  claimed  to  be  approximately  log  normal.  However,  since  no  co- 
gent statistical  evidence  for  this  result  has  been  found,  it  is  be- 
lieved that  a more  realistic  interpretation  of  the  observed  data  is 
that  the  distribution  of  Delphi  responses  is  unimodal  and  skewed  to 
the  right,  a behavior  characterizing  not  only  log  normal  distributions 
but  many  others,  such  as  the  gamma,  as  well. 

Should  the  sets  of  responses  to  Delphi  exercises  be  treated  dif- 
ferently from  one  another,  depending  on  the  nature  of  the  questions 
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being  raised?  There  is  some  evidence  that  they  should.  Three  basic 
types  of  questions  can  be  distinguished:  First,  there  is  the  "almanac" 

type  of  question,  which  has  an  easily  established  answer,  but  most 
people  are  not  likely  to  know  it.  For  verification,  we  might  look  for 
the  answer  in  an  almanac,  or  we  might  make  direct  measurements  or  ob- 
servations (such  as  in  the  problem  of  guessing  the  weight  of  an  apple). 
A second  type  of  question  involves  group  value  judgments,  such  as  de- 
termining the  desirable  quality  of  various  aspects  of  life,  or  the 
desirable  level  of  mean  education  that  society  should  seek  for  its 
people.  Dalkey  and  Rourke  conclude  that  "Delphi  procedures  are  appro- 
priate for  processing  value  material  as  well  as  factual  material"  (I 
believe  this  conclusion  to  be  overdrawn).  The  third  type  of  question 
requires  the  subject  to  make  some  predictions  or  forecasts.  Dalkey 
and  Brown  carried  out  an  experiment  involving  both  almanac  questions 
and  short-range  predictions  (less  than  6 months).  They  conclude  that 
"no  significant  difference  was  observed  between  these  two  kinds  of 
estimates."  Again,  there  is  reason  to  suspect  such  a conclusion,  since 
the  basis  for  such  a significance  test  has  not  been  indicated  (was  a 
log  normal  distribution  of  responses  assumed,  etc.?). 

CATEGORICAL-DEPENDENT-VARIABLE  MULTIVARIATE  REGRESSION 

In  this  section  we  will  present  a simplified  version  of  a new 
methodological  procedure,  developed  by  Nerlove  and  Press,  for  doing 

^ See,  for  example,  Dalkey,  The  Delphi  Method * An  Experimental 
Study  of  Group  Opinion . 

^See,  for  example,  Dalkey  and  Rourke,  op.  cit.,  and  Dalkey  and 
Brown,  op.  cit. 
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regression  analysis,  using  categorical  dependent  variables.  We  will 

show  how  the  technique  can  be  applied  to  the  problem  of  assessing 

technological  advance  and  of  comparing  the  feasibility  of  competing 

R&D  projects.  The  methodology  generalizes  the  results  of  categorical- 

dependent-variable-regression,  single-equation  systems  to  correlated, 

multiple-equation  systems  of  the  same  form.  An  extensive  treatment  of 

the  subject  is  given  in  the  appendix,  and  empirical  applications  of 
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the  method  are  presented  elsewhere. 

Suppose  there  is  a panel  of  n subjects,  each  member  of  which  is 
asked  to  judge  technological  advancement  for  N projects,  S^,...,S  . 
Define  the  endogenous  (dependent)  indicator  variables,  y^ ^ (k) , as 
follows : 


yij  00 


1,  if  event  E^  (k)  occurs, 
0,  otherwise. 


where  E (k)  denotes  the  event  that  in  a pairwise  comparison,  the  kth 
subject  assesses  the  degree  of  technological  advancement  of  to  be 
greater  than  that  of  S : i,  j = 1,...,N;  i i j;  k = l,...,n. 

Next  suppose  the  exogenous  (independent)  variables  X^_., 

X2ij> • • • »xrij  kear  directly  upon  whether  project  is  more  techno- 
logically advanced  than  S^.  Let  x^_  (k)  >x21j  (k)  » • • • ,xrij  (k)  denote 
the  values  perceived  for  , . . . >xrij  bY  tbe  kth  subject.  These 
values  really  form  the  basis  for  the  kth  subject’s  assessments  and  may 
be  thought  of  as  quantitatively  defining  the  set  of  assumptions  he 
makes  when  he  renders  his  comparative  judgments. ^ 

Adopt  the  model 


yij (k)  = F[g0ij  + BlijXlij(k)  + + prijxrij(k)]  + Uij(k)’ 
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M.  Nerlove  and  S.  J.  Press,  Univariate  and  Multivariate  Log - 
Linear  and  Logistic  Models , The  Rand  Corporation,  R-1306-EDA/NIH, 
December  1973. 
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For  example,  the  x’s  may  measure  the  degree  of  information  the 
kth  subject  has  about  the  relative  difficulty  between  two  projects, 
or  the  depth  of  background  or  experience  the  kth  subject  has  relative 
to  projects  i and  j. 
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where  u (k)  denotes  a random  disturbance  term  with  the  properties  that 
Ef^ijCk)]  = 0,  and  E[u_  (k1>u_  (k2)  ] = 0,  k^  ± k^;  and  F(*)  denotes  a 
monotonic  nondecreasing  transformation  with  0 < F(0  < 1;  i,  j * 

k - l,...,n.  The  function  F(#)  will  be  taken  to  be  the  cumu- 
lative distribution  function  of  the  logistic  distribution,  F(t)  = 

[1  + e *“]  \ -°°  < t < 00 . The  rationale  for  this  choice  is  given  in 
the  appendix. 

For  algebraic  simplicity,  define  the  r-dimensional  column  vectors 


and 


3ij  E (30ij’--*’3rij)V 


x±j(k)  = [1,  xlij(k),...,xrlj(k)]', 


so  that  the  model  becomes 


(k)  = Flx’jCkjB^]  + u±j  (k) » 


for  i,  j = 1,2,...,N;  k = l,...,n.  In  any  particular  problem  we  assume 
that  x (k)  is  given  (subject  k may  specify  it  in  part  or  in  toto), 
and  then  y (k)  is  generated  by  assessment  (when  subject  k renders  his 
judgment  about  vis-&-vis  S^). 

Define 


Pj. j (k)  E p{y±j(k)  - 1}  " P{Ei:j(k)}. 


That  is,  since 


Pj.j  (k)  = E[y±j  (k>]> 

pij(k)  = Ffx^k)^], 
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i,  j = 1,...,N;  k = 1,  . ..,n.  Thus,  by  using  the  sample  of  n subjects 


to  estimate  3..  as  3..,  p*#,  the  probability  that  S.  is  more  techno- 
ij  13  11  i 

logically  advanced  than  S_.,  given  any  preassigned  set  of  assumptions, 
is  estimated  as 


ns  2 


1 > 3 ~ 1 » • • • 


But  the  disturbance  terms  u^ (k)  are  mutually  correlated  for  various 
i and  j,  for  a fixed  k (even  though  they  are  assumed  uncorrelated  for 
different  kfs),  so  the  (k)  are  mutually  correlated  for  fixed  k. 

Hence,  there  is  information  in  one  equation  that  can  be  used,  in  part, 
to  estimate  parameters  in  other  equations.  That  is,  the  system  of 
equations  should  be  viewed  simultaneously  as  a set  of  multivariate  non- 
linear regression  equations  in  which  the  endogenous  variables  (the  yfs) 

are  reflective  of  a categorical  probability  and  in  which  there  is  sys- 
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tematic  heteroscedasticity . The  solution  to  this  problem  is  a set 
of  estimates  of  p*_.,  for  i>  j = 1,...,N,  telling  the  analyst  the  rela- 
tive degrees  of  technological  advance  required  for  a set  of  R&D  projects 
(useful,  for  example,  in  cost  equations)  or  telling  the  policymaker 
how  to  view  the  level  of  difficulty  of  a new  project.  The  statistical 
method  devised  for  solving  this  problem  is  given  in  detail  in  the  ap- 
pendix. 

Suppose,  for  example,  there  are  three  systems  to  be  compared  re- 
garding technological  advancement.  Recall  that  y^^ (k)  was  defined  to 
be  one  or  zero,  depending  on  whether  or  not  in  a pairwise  comparison 
the  kth  subject  assesses  the  degree  of  technological  advancement  of 

,n. 

Take  N = 3 (three  projects  to  be  compared)  and  define  the  random  vari- 
able for  which  y (k)  is  the  kth  observed  value.  The  three  random 

variables,  Y^,  Y^»  Y239  are  mutual^Y  correlated  and  completely  de- 
scribe the  comparative  states  of  advancement  of  the  three  systems  (note 

that  if  ties  are  ruled  out,  Y , . = 1 - Y. „ so  that  random  variables 

13 


to  be  greater  than  that  of  S^,  i,  j = 1, . . . ,N,  i 4 j>  k = 1, 


41 


Unequal  variances  of  the  disturbance  terms  for  a given  k. 
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Y for  which  j < i are  unnecessary).  Now  rename  the  variables, 

Y1  " Yi2*  Y2  E Y13>  Y3  e Y23*  111611  > taking  Y^  « 8^  + j » 1,  2,  3, 

where  denotes  an  error  term  with  mean  zero,  puts  the  problem  into 
a three-equation  model.  The  B^'s  will,  of  course,  be  taken  to  be  the 
monotone  transformations  of  linear  combinations  of  independent  varia- 
bles discussed  above.  In  this  form  the  system  can  be  thought  of  as  a 

3 

2 or  2x2x2  (trivariate)  contingency  table  that  can,  in  turn,  be  thought 
of  as  a trivariate  categorical-dependent-variable  regression  equation 
in  which  the  dependent  variable  can  assume  eight  possible  values.  If 
N systems  are  to  be  compared,  there  will  be  N(N  - l)/2  simultaneous 
equations  to  be  solved  in  this  way,  rather  than  the  three  used  in  the 
example . 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


We  have  seen  how  diverse  procedures  developed  in  different  dis- 
ciplines might  be  brought  to  bear  on  the  problem  of  how  to  combine  the 
opinions  of  individuals  to  form  a group  judgment  about  an  ill-defined, 
multidimensional  concept,  such  as  the  degree  of  technological  advance 
required  to  complete  a given  R&D  project,  or  the  probability  that  a 
certain  technological  development  will  become  feasible  by  a preassigned 
date.  Although  there  are  advantages  with  each  procedure  suggested, 
there  are  also  various  difficulties,  uncertainties,  and  limitations, 
both  conceptually  and  technically.  The  techniques  described  may  also 
vary  in  their  suitability  for  application  according  to  the  type  of 
R&D  management  practices  presently  in  use. 

In  summary,  the  methods  of  individual-differences  scaling,  sub- 
jective probability,  and  categorical-dependent-variable  multivariate 
regression  are  very  attractive  for  quantification  and  analysis  of 
group  judgment  data.  A controlled  feedback  approach  could  be  useful 
in  carrying  out  both  an  individual-differences  scaling  and  a categorical- 
dependent-variable  multivariate  regression.  That  is,  a panel  of  ex- 
perts might  evaluate  the  relative  merits  of  a collection  of  R&D  proj- 
ects, perhaps  both  completed  and  projected.  In  the  case  of  individual- 
differences  scaling,  group  opinions  can  be  fed  back  in  a multistage 
approach  to  produce  a scaled  solution  at  each  stage,  because  the  cur- 
rent state  of  the  methodology  is  strictly  mathematical  and  has  not  yet 
advanced  to  the  point  where  statistical  inferences  can  be  drawn. 

In  categorical-dependent-variable  multivariate  regression,  there 
are  advantages  in  being  able  to  relate  the  experts’  responses  to  their 
backgrounds,  their  degrees  of  expertise,  and  their  definitions  of  the 
"difficulty”  parameters  of  each  project;  also,  statistical  inferences 

and  predictions  are  possible.  However,  at  least  at  the  present  time, 
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the  method  should  not  be  used  with  feedback.  Nevertheless,  the 
42 

By  using  a multistage  procedure  that  tells  each  subject  at  each 
stage  what  the  group  opinions  were  at  the  previous  stage,  we  are  in 
effect  generating  intrinsic  collusion  among  the  subjects.  The  effect 
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methodology  may  be  used  to  advantage  to  analyze  the  first-stage  results 
of  a controlled  feedback  process. 

Results  derived  from  the  assessment-data  methods  of  analysis  de- 
scribed above  could  be  used  to  develop  predictive  models  for  determin- 
ing the  costs  and  feasibility  probabilities  of  proposed  and  projected 
projects.  Moreover,  it  should  be  possible  to  relate  the  cost  and 
probability  predictions  to  the  assumptions  underlying  the  predictions. 

In  conclusion,  methods  should  be  refined  for  selecting  a panel  of 
appropriate  experts  to  provide  assessments  of  technological  advancement 
or  feasibility  of  a collection  of  R&D  projects.  Planning  objectives 
should  be  devised  for  selecting  technology  areas,  and  criteria  should 
be  established  for  determining  the  number  and  type  of  experts.  If  a 
set  of  projects  is  to  be  studied  and  evaluated,  a controlled  feedback 
type  of  analysis  should  first  be  pretested  on  the  panel  of  experts. 

An  interrogation  procedure  should  be  developed,  with  the  questions  de- 
signed to  assess  technological  advancement  in  the  specific  technologies. 
Computer  programs  should  be  obtained  or  developed  (if  they  are  not 
accessible)  for  analyzing  data  by  means  of  both  individual-differences 
scaling  and  categorical-dependent -variable  multivariate  regression. 

At  least  two  types  of  analysis  should  be  undertaken:  individual- 

differences  scaling  using  a multistage  controlled  feedback  approach, 
and  a controlled  feedback-based,  categorical-dependent-variable  multi- 
variate regression  but  without  the  feedback  data. 

of  such  collusion  upon  the  statistical  analysis  is  to  violate  the  as- 
sumption of  independence  among  the  subjects1  responses.  At  the  pres- 
ent time,  the  model  formulation  previously  described  will  not  permit 
correlated  observations  (interdependent  response  vectors).  Hence,  the 
results  of  a feedback  process  should  not  be  analyzed  in  this  way. 
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Appendix 

THEORY  OF  LOG-LINEAR  AND  LOGISTIC  MODELS  FOR  THE  ANALYSIS  OF 
QUALITATIVE  JUDGMENTS  OF  TECHNOLOGICAL  ADVANCEMENT 

Marc  Nerlove  and  S.  James  Press 


A.l.  INTRODUCTION 


In  his  excellent  survey  of  the  ideas  that  have  been  presented  about 
inference  in  contingency  tables,  Mosteller  (1968)  stated,  "I  fear  that 
the  first  act  of  most  social  scientists  upon  seeing  a contingency  table 
is  to  compute  chi-square  for  it.  Sometimes  this  process  is  enlighten- 
ing, sometimes  wasteful,  but  sometimes  it  does  not  go  quite  far  enough.11 
In  this  appendix  we  will  summarize  the  most  central  earlier  efforts, 
to  give  us  an  understanding  of  such  cross  tabulations,  and  then  we  will 
present  some  new  results  that  promise  to  push  our  understanding  a bit 
further. 

This  sequel  is  concerned  with  estimating  relationships  that  deter- 
mine jointly  dependent  qualitative  variables,  i.e.,  variables  that  are 
both  categorical  and  unordered.  For  example,  a military  system  requir- 
ing sophisticated,  advanced  technology  may  or  may  not  be  feasible  by 
a preassigned  date.  We  may  wish  to  relate  the  probability  that  the  new 
system  will  be  feasible  by  that  date  to  several  explanatory  variables, 
such  as  the  rate  of  funding  for  the  development  of  the  required  method- 
ology, the  probabilities  that  required  subsystems  will  be  available 
early  enough,  etc. 

We  will  first  provide  a brief  review  of  the  single  qualitative 
variable  problem.  Section  A. 2 treats  the  dichotomous,  single  qualita- 
tive variable  case,  and  Section  A. 3 discusses  the  poly tomous , single 
qualitative  variable  problem.  Section  A. 4 generalizes  the  known  re- 
sults for  a single  qualitative  variable  ("univariate"  case) , treated 
earlier,  to  some  new  results  for  several  jointly  dependent  qualitative 
variables  ("multivariate"  case).  Results  are  generally  applicable  to 
a wide  variety  of  problems  involving  the  estimation  of  probabilities 
associated  with  R&D  planning  functions. 

Equation  numbers  and  figure  numbers  appearing  in  the  appendix  will 
always  refer  to  those  equations  and  figures  appearing  in  the  appendix. 
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A.2.  ONE  DICHOTOMOUS  QUALITATIVE  VARIABLE 


FAILURE  OF  THE  STANDARD  REGRESSION  MODEL  AND  ATTEMPTS  AT  FIRST  AID 

Let  y denote  a dichotomous  random  variable  that  is  one  if  event  E 
occurs,  and  zero  otherwise,  and  let  x:  p x 1 denote  a p-vector  of  vari- 
ables used  as  explanatory  or  predictor  variables  for  y (they  may  actu- 
ally be  transformations  of  more  basic  variables).  Suppose  (Yj * xj) » 
j = l,...,n  are  independent  observations  of  (y,  x) . Some  of  the  expla- 
natory variables  may  themselves  be  categorical  and  some  may  vary  con- 
tinuously. Let  g:  p x 1 denote  a set  of  unknown  weights  so  that  x'g 
specifies  a linear  function  of  x (the  prime  denotes  transpose) . Adopt 
the  standard  regression  model 

y = x*  3 + j = l,...,n  , (1) 

where  e denotes  a disturbance  term  characterized  by  the  distributional 

2> 

properties  that  E(e.)  = 0;  var(e^)  = c ; E(eje^)  = 0,  for  j 4 k,  k 
1, • • . ,n»  If  theory  demands  that  the  larger  the  value  of  xf8  the  greater 
the  chance  that  y will  be  one,  we  can  think  in  terms  of  a monotone 
relationship  between  xT8  and  the  probability  of  event  E (see  Fig.  A. 1) . 
The  true  probability  function  should  generally  have  the  characteristic 
sigmoid  or  S shape  of  a cumulative  distribution  function  (cdf)  since 
it  must  lie  between  zero  and  one  and  be  nondecreasing.  Thus,  if 
p 5 P[E],  p (which  depends  on  xT8)  is  plotted  as  a function  of  x 8 as 
the  solid  curve  in  Fig.  A.l.  We  have  drawn  the  true  curve  so  that  its 
upper  and  lower  asymptotes  are  indeed  one  and  zero.  Although  the  func- 
tion can  be  well  approximated  in  the  center  of  its  domain  by  the  uncon- 
strained dashed  line  segment  AB,  the  unconstrained  straight  line  is  a 
poor  approximation  for  very  large  or  very  small  values  of  x 8 , and, 
indeed,  violates  the  condition  that  the  function  lie  between  zero  and 
one  for  extreme  values  of  the  argument.  A possible  solution  is  to  re- 
place the  dashed  line  by  a broken  line  approximation  that  consists  of 
the  dashed  line  between  the  asymptotes  and  the  asymptotes  themselves 
above  and  below  those  values  of  xT8  for  which  the  dashed  line  lies 
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outside  the  appropriate  range.  As  can  be  seen  from  Fig.  A. 1,  the  broken 
line  approximation  is  a good  one  except  for  two  small  regions  near  the 
points  where  the  asymptotes  are  crossed.  Of  course,  the  size  of  such 
regions  of  poor  approximation  will  vary  depending  on  the  shape  of  the 
true  probability  function. 

Fitting  a broken  line  approximation  to  binary  observations  result- 
ing from  a nonlinear  probability  function  is  no  easy  matter.  If  one 
had  many  observations  of  y for  each  value  of  x,  as  is  typical  in  bioas- 
say (see  the  next  section),  a number  of  solutions  would  be  possible. 
Suppose,  however,  that  there  is  only  one  observation  of  y for  each  value 
of  x.  This  value  may  conveniently  be  called  zero  or  one.  Figure  A. 2 
illustrates  the  effect  of  imposing  the  appropriate  constraint  on  a 
broken  line  approximation.  The  observations  are  indicated  by  small 
circles  lying  on  the  upper  or  lower  asymptotes  of  one  and  zero.  It  is 
possible  to  fit  the  data  exactly,  provided  the  observations  do  not 
force  a negative  slope  on  the  interior  segment.  Indeed,  the  line  of 
perfect  fit  is  not  even  unique.  If  we  were  to  impose  the  constraint 
that  the  interior  segment  be  nonnegatively  sloped,  in  the  limit  as  the 
number  of  observations  became  very  large,  some  values  of  y = 1 and  of 
y 0 would  occur  for  values  of  x in  inverted  order,  so  the  central 
Por^^on  the  optimal  broken  line  segment  would  be  vertical!  In  the 
mu^*::*'var^'a*:e  case>  difficult  computational  procedures  would  be  required 
to  fit  a broken  line  approximation,  but  computational  difficulties  are 
the  least  of  the  undesirable  features  of  such  approximations. 

Such  difficulties  have  led  many  econometricians  (see  especially 
Goldberger,  1964,  p.  249)  to  suggest  the  use  of  linear  regression  with 
the  binary  variable  y as  dependent.  There  are  a number  of  important 
difficulties  with  estimating  g in  (1)  by  least  squares.  First  note 
that  under  the  assumptions  of  the  model  in  (1) , for  fixed  x , y is  a 
Bernoulli  random  variable  so  that  E(y.lx.)  = x!g  and  var(y  lx  ) = 

j J j *J  j 

Var^£j^  ~ ""  Xj^*  Since  var(ej)  depends  upon  j,  the  are  hetero- 

scedastic,  and  the  use  of  ordinary  least  squares  estimation  will  gener- 
ate inefficient  estimators  and  imprecise  predictions.  Note  also  that 
Xjg  can  have  any  value  on  the  real  line,  whereas  y^ , and  therefore 
E(yj|x^),  is  restricted  to  the  unit  interval.  As  a result,  predictions 
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lying  outside  the  unit  interval  may  sometimes  be  produced  for  small  or 
large  values  of  x'B.  Note  also  that  the  fitted  relationship  is  excep- 
tionally sensitive  to  the  location  of  the  explanatory  variables  and  that 
the  usual  tests  of  significance  for  the  estimated  coefficients  do  not 
apply.  Further,  multiple  R no  longer  is  meaningful,  and  estimated  stan- 
dard errors  are  not  consistent.  Finally,  we  note,  with  Cox  (1970,  p,  17), 


that  "because  the  y^'s  are  not  normally  distributed,  no  method  of  estima- 
tion that  is  linear  in  the  y ^ 1 s will  in  general  be  fully  efficient. 

That  is,  we  can  always  improve  on  the  least  squares  estimation  (whether 


or  not  it  is  corrected  for  heteroscedasticity)  since  it  is  a linear 


estimator. 

In  an  experimental  situation,  one  would  have  control  over  many  of 
the  explanatory  variables  and  could,  and  normally  would,  ensure  that 
the  values  of  these  variables  covered  a wide  range  and  were  not  abnor- 
mally bunched  at  high  or  low  levels.  In  a nonexperimental  situation, 
however,  we  are  unlikely  to  be  so  fortunate  as  to  have  our  explanatory 
variables  distributed  more  or  less  uniformly  over  a wide  range. 

It  has  been  suggested  that  the  heteroscedasticity  problem  can  be 
removed  by  using  generalized  least  squares  (see  Goldberger,  1964,  p. 
248;  Zellner  and  Lee,  1965,  p.  387),  Unfortunately,  this  approach  can 
be  relied  upon  only  asymptotically.  Moreover,  except  for  making  use 
of  the  first  two  moments  of  z ^ it  completely  ignores  the  Bernoulli 
character  of  the  errors. 

To  correct  for  the  problem  of  heteroscedasticity,  Goldberger  sug- 
gests replacing  the  variances  of  the  z ^ in  a generalized  least  squares 
analysis  by  the  estimates  y ^ (1  - y^),  where  the  y are  the  calculated 
values  of  the  y from  an  ordinary  least  squares  regression  of  y^  on  x^. 

Unfortunately,  there  is  no  guarantee  that  y will  lie  between  zero  and 

1 

one  for  all  j,  so  some  of  the  "variances"  may  be  negative. 

Smith  and  Cicchetti  (1972)  have  done  extensive  Monte  Carlo  studies 
on  alternative  methods  of  handling  inadmissible  weights  from  a first 


HlcGillivray  (1970)  shows  that  (1  - y^)  is  a consistent  estimator 
of  (Ey^)(l  - Ey^),  although  the  former  may  be  negative  but  the  latter 


cannot  be. 
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stage  OLS  regression  in  a generalized  least  squares  analysis.  These 
are:  (1)  setting  y^  - 0.5  when  y^  (1  - y^)  <0;  (2)  replacing  -y  (1  - y ) 
by  |Yj (1  ~ Yj)|;  and  (3)  replacing  y^  by  0.98  when  y (1  - y ) < 0. 

These  three  generalized  least  squares  estimators  are  compared  with  OLS 
for  samples  of  size  20  and  size  100  for  regressions  involving  both  dicho- 
tomous and  continuous  explanatory  variables  and  several  alternative  sets 
of  true  coefficients.  The  results  of  these  experiments  are  difficult 
to  summarize  except  that  the  larger  sample  evidently  reduces  the  ap- 
parent differences  among  all  methods.  All  the  estimates  are  biased, 
but  the  authors  do  not  seem  to  have  examined  the  relation  of  the  dis- 
persion of  the  explanatory  variables  to  the  amount  of  bias  present. 

The  estimated  variances  of  the  estimates  are  badly  biased  toward  zero 
and  the  power  of  the  Student’s  t-test  for  testing  hypotheses  with  the 
estimated  coefficients  is  very  limited.” 

Generating  estimates  and  predictions  that  lie  outside  the  unit 
interval  can  be  avoided  by  doing  "restricted”  least  squares  estimation 
in  which  the  estimators  are  constrained  to  the  unit  interval  in  the 
minimization.  Especially  when  heteroscedasticity  is  accounted  for, 
the  resulting  computation  is  particularly  complex,  it  is  reliable  only 
in  large  samples,  and  it  ignores  most  of  the  distributional  properties 
of  the  errors.  Figure  A. 3 illustrates  the  sensitivity  of  an  unrestricted 
least  squares  fit  of  a linear  probability  function  (Eq.  (1))  to  bunched 
data.  When  most  values  of  xf8  are  large,  the  preponderance  of  observa- 
tions on  y will  be  1;  when  most  of  the  values  of  xfB  are  small,  the 
preponderance  will  be  0.  The  location  and  slope  of  the  fitted  rela- 
tionship may  be  greatly  affected.  In  an  extreme  situation  (when  all 
values  of  xf6  are  zero  or  all  are  one),  the  fitted  relationship  may 
have  zero  slope  and  no  intersection  with  the  ordinate  between  zero  and 
one.  No  method  of  estimation  can  possibly  function  well  in  such  extreme 
cases,  but  the  linear  probability  function  is  more  likely  to  be  subject 
to  such  difficulties  in  non-ideal  situations  than  a method  more  closely 
approximating  the  true  probability  function.  Misspecification, is  bad; 
bunched  data  are  generally  bad;  combining  one  with  the  other  can  only 
aggravate  the  difficulties. 
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Fig.  A. 3 — Illustration  of  the  sensitivity  of  Ordinary  Lease  Squares  (OLS) 
to  the  values  of  the  explanatory  variable 
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P ROB IT  ANALYSIS  (MANY  OBSERVATIONS  PER  CELL) 

It  was  pointed  out  above  that  applications  of  standard  regression 
techniques  to  binary  data  has  many  technical  difficulties.  One  of  the 
earliest  methods  of  analyzing  such  data,  "probit  analysis,"  has  been 
attributed  to  J.  H.  Gaddum  and  C.  I.  Bliss  by  Ashton  (1972).  Some  of 
this  early  work  was  described  in  Bliss  (1934) . Finney  (1947)  applied 
probit  analysis  to  the  problem  of  analyzing  quantal  (binary)  responses 
in  bioassay. 

The  bioassay  problem  is  one  of  studying,  for  example,  the  potency 

of  some  drug  by  observing  the  behavior  of  animals  to  whom  various  doses 

of  the  drug  have  been  administered.  Suppose  each  animal  in  a group  of 

n animals  is  given  a dose  t of  a poisonous  drug,  j = 1,...,G;  let 

J 

yii  be  unity  if  animal  i in  group  j dies,  and  let  y be  zero  other- 

-*-j 


wise,  i = 1, . . . ,n 


j' 


Let  Pj  denote  the  probability  that  an  animal  in 


group  j will  die  from  the  drug,  and  let  (a  + bt^)  denote  the  "stimulus 
level"  of  the  drug,  where  a and  b are  unknown  constants.  If  one  wants 
a relationship  in  which  p^  is  a nondecreasing  function  of  t ^ , the  model 


Pj  = F (a  + btj) 


where  F(t)  denotes  a cumulative  distribution  function,  may  be  used; 
so  that  automatically  0 j<  p^  1 and  p^  is  a nondecreasing  function  of 


t_j  for  b > 0. 


More  generally,  let  x j 8 denote  the  stimulus  level,  where 


x^ : p x 1 denotes  a p-vector  of  stimulus  variables  and  8 denotes  a p- 
vector  of  weights.  Then  the  model  is 


Pj  * F^xj^  * j « 1, . . . ,G  , 


(2) 


where  p^  is  nondecreasing  in  xjB,  but  may  be  decreasing  in  some  stimulus 

variables,  depending  upon  the  algebraic  signs  of  the  components  of  8 

(the  elements  of  x may  be  transformations  of  some  more  basic  variables). 

Probit  analysis  describes  a collection  of  methods  for  estimating 

2 

the  relationship  in  (2) , using  grouped  data,  when  F(t)  is  taken  to  be 


The  term  "grouped  data"  is  used  throughout  synonymously  with  the 
term  "many  observations  per  cell,"  to  mean  that  there  is  more  than  one 
observed  y (dichotomous  or  polytomous)  for  each  observed  x-vector. 


sam- 
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the  cdf  of  the  standardized  normal  distribution.  First  define  the 
pie  estimator  of  p^ , 


~ _ 1 J 

Pj  - ^ yj4t  J = , 

J i=1  ij 


(3) 


which  is  seen  to  be  a reasonable  estimator  since  p^  = p{y_^  = 1};  it 
is  also  a maximum  likelihood  estimator.  Next  define 


z.  = Probit  OPj)  + 5 , 


(4) 


where  ^ is  defined  by  = F(t.);  that  is. 


PJ  = 


* f' 

J 


-l/2t 

e dt 


The  5 is  added  to  the  fractile  t^  in  (4)  to  avoid  negative  values  for 
Zj * Finally,  if  we  adopt  the  model 


zj  = x?  + ey  j - , 


(5) 


where  E(£j)  = 0,  var(£j)  « a2,  E(ej£k)  * 0,  j i k,  6 in  (5)  may  be 
estimated  by  ordinary  least  squares.  For  this  probit  analysis  method 
to  be  useful,  there  should  be  several  observations  per  cell  (n^  > for 
every  j).  Moreover,  efficiency  of  estimation  is  lost  in  the  ad  hoc 
procedure  associated  with  the  added  5 in  (4) . Note  also  that  there 
are  computational  difficulties  associated  with  the  use  of  the  integrals 
in  this  procedure,  unequal  numbers  of  observations  per  cell  are  ineffi- 
cient, and  cells  with  one  or  zero  observations  per  cell  are  not  useful. 

The  use  of  probit  analysis  in  econometrics  and  the  social  sciences 
has  not  been  widespread.  Economic  and  sociological  data  are  generally 
nonexperimental  (biological  data  are  usually  the  result  of  designed 

experiments) , and  it  has  until  recently  been  rare  to  collect  more  than 

3 

one  observation  per  cell.  In  such  cases,  frequency  estimates  of  the 


For  an  exception,  however,  see  Tobin  (1955). 
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type  given  in  (3)  have  not  been  possible.  Now,  however,  increasing 
availability  of  survey  data  for  individuals,  households,  or  firms  makes 
methods  that  rely  upon  frequency  estimates  more  acceptable  and  more 
usable  than  ever  before.^ 


LOGIT  ANALYSIS  (MANY  OBSERVATIONS  PER  CELL) 

Logit  analysis  is  a term  coined  by  Berkson  (1944,  1953,  1955)  to 
characterize  methods  of  estimating  the  relationships  in  (2),  using 
grouped  data,  when  F(t)  is  taken  to  be  the  cdf  of  the  standardized 
logistic  distribution;  that  is, 

F(t)  = i — --  , - oo  < t < oo  , (6) 

1 + e’11 


Combining  (2)  and  (6)  gives 


1 + e 


-x*3 


(7) 


Solving  for  the  argument  gives 


Berkson  defined  the  estimated  log-odds, 


z 


j 


3 Logit  (Pj)  > 


(8) 


(9) 


where  p^  is  defined  in  (3) . Now  adopt  the  model  of  (5)  and  estimate 
6 by  least  squares  (note  that  p^  cannot  be  zero  or  one  in  (9) , which 
would  occur  if  nj  s 1).  It  is  well  known  that  the  numerical  difference 
between  the  normal  and  logistic  cdf  is  very  slight  except  at  either  of 


— This  point  has  been  stressed  much  earlier  by  Orcutt,  Greenberger, 
Korbel,  and  Rivlin  (1961) , 
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the  extremes  (see  Cox,  1970,  p.  28;  or  Ashton,  1972,  p.  11).  The  logit 
analysis  method  has  the  advantage  of  numerical  simplicity  over  probit 
analysis,  but  the  estimation  method  using  logits  still  requires  the 
availability  of  grouped  data. 

The  generalized  least  squares  method  suggested  by  Theil  (1970,  p. 
109)  for  analyzing  binary  response  data  for  a single  qualitative  vari- 
able deletes  cells  containing  only  one  or  no  observations  and  is 
aPplfc^ble  only  in  large  samples.  Unfortunately,  in  an  economic  con- 
text it  is  easy  to  envision  situations  in  which  every  cell  will  have 
only  one  or  no  observations  (for  example,  when  one  of  the  explanatory 
variables  is  not  under  control  and  is  continuous)  and  in  which  the 
sample  is  small.  A continuous  variable  can  always  be  categorized,  but 
unless  the  total  number  of  observations  is  very  large,  much  informa- 
tion may  be  lost.  It  is  still  desirable  to  develop  methods  that  are 
directly  applicable  to  one  observation  per  cell  and  will  have  credible 
small  sample  properties.  The  method  of  maximum  likelihood,  described 
below,  has  such  properties  and  was  applied  to  this  problem  at  a very 
early  stage.5 


An  alternative  method  of  estimation  called  "minimum  Chi  square" 
was  devised  by  Berkson  (1955) . The  results  are  asymptotically  identi- 
cal to  maximum  likelihood  estimation  (MLE)  but  in  large  samples  are 
simpler  to  evaluate.  The  procedure  is  developed  from  the  idea  of  mini- 
mizing the  conventional  Chi  square  statistic  for  a multinomial  table, 


2 

X 


G 

I 

j=l 


“jtPj  - F(Xj 8) ]2{F(Xj6)  [1  - F(xp)]}-1 


with  respect  to  3.  The  quantities  and  F(t)  are  defined  in  (3)  and 

(6),  This  result  is  equivalent  to  weighting  the  squared  deviations 
between  observed  and  expected  numbers  of  observations  by  the  recipro- 
cals of  the  variances  of  the  observations.  When  large  samples  are 
present,  this  expression  may  be  linearized  to  yield  very  simple  solu- 
tions for  the  elements  of  3.  When  all  y in  a cell  are  zero  or  one, 

however,  the  results  must  be  modified  appropriately  so  as  not  to  lose 
this  information.  In  any  case,  the  minimum  Chi  square  procedure  re- 
quires more  than  one  observation  per  cell  (preferably  many  observa- 
tions per  cell) . 
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The  logistic  transformation  has  been  justified  for  studying  rela- 
tionships of  the  form  given  in  (2)  on  the  basis  that  it  is  simple  and 
tractable  algebraically,  and  therefore  simple  computationally,  and  its 
cdf  is  numerically  close  to  the  normal  cdf.  These  facts,  though  useful 
and  interesting,  can  hardly  be  considered  adequate  justification  for 
the  arbitrary  selection  of  a functional  form  for  F(t) . In  a more 
fundamental  justification,  Truett,  Cornfield,  and  Kannel  (1967)  pro- 
vide a physical  mechanism  for  this  useful  transformation.  Although 
their  work  was  carried  forth  in  the  setting  of  a disease  and  its  asso- 
ciated syndrome,  the  ideas  apply  generally. 

Let  y = 1 if  event  E occurs,  and  y = 0 if  E does  not  occur — that 
is,  if  E occurs.  Then,  if  x is  a p-vector  with  continuous  density 
h (x | 0) , where  0 is  a parameter  matrix  indexing  the  distribution,  by 
Bayes  theorem, 


P(E  x) 


P{E}h(xl E,9) 

P{E}h(x|E,0)  + P{E}h(x| E,0) 


Let  q = P{E},  and  let  p = P{E|x}  = P{y  = l|x}.  Then, 


P = 


1 + 


h(x 

E,e) 

h(x 

E,e) 

Now  suppose  that  given  (E , 0) , d£(x)  = N(0- , E)  ; and  given  (E,  0), 

6 

d£(x)  = N(6  , E).  Then,  it  is  straightforward  to  check  that  p may  be 
written  in  the  form 


P = 


1 4*  e 


-xf  3 


for  some  vector  of  weights  3 that  depend  upon  0^,  0^*  and  E;  i.e.,  p 
follows  the  logistic  cdf  form. 


csf(x)  = N(0,  E)  implies  that  the  probability  law  of  x is  normal 
with  mean  0 and  covariance  matrix  E. 
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It  may  be  argued  that  in  many  situations  x does  not  follow  a multi- 
variate normal  distribution.  Halperin,s Blackwelder , and  Verter  (1971) 
compared  conditional  MLEs  for  this  model,  assuming  some  of  the  x were 
dichotomous,  with  discriminant  function  estimates  obtained  assuming 
normality  of  the  x (the  latter  are  unconditional  MLEs).  Their  results 
showed  that  when  normality  of  the  x was  violated,  ML  estimators  still 
behaved  appropriately  (and  gave  somewhat  better  fits  to  the  model  than 
discriminant  function  estimators) . For  this  reason,  as  well  as  for 
those  mentioned  earlier,  we  restrict  attention  to  logistic  transforma- 
tions of  the  data. 


OTHER  TRANSFORMATIONS  WITH  MANY  OBSERVATIONS  PER  CELL 

Transformations  other  than  the  normal  and  the  logistic  have  been 
proposed  as  appropriate  functions  in  (2)  for  studying  relationships 
like  those  discussed  in  this  report.  Several  of  them,  for  example, 

F^(t)  = 1/2(1  + sin  t},  —rr/2  < t < tt/2  , 

F0(t)  = 1/2  + — arctan  t,  -»  < t < 00  , 

. Z 7T 

F^(t)  = tanh  t/2,  -»  < t < °°  , 


are  often  grouped  together  under  the  designation  ’'angular  transforma- 
tions." They  are  numerically  similar  to  the  normal  curve,  except  in 
the  tails.  Moreover,  in  some  econometric  problems  where  there  is  pri- 
mary interest  in  one  or  both  of  the  tails  of  the  distribution  (rare 
or  very  frequent  events)  an  angular  transformation  may  prove  to  be 
more  appropriate  than  a logit  or  probit.  In  such  a case  the  parameters 
of  the  model  may  be  estimated  by  least  squares,  provided  that  there 
are  many  observations  per  cell.  (If  not,  then  maximum  likelihood 
methods  similar  to  those  developed  in  this  report  must  be  formulated.) 

For  example,  using  F^(x^3) , if  z ^ = arcsin  (2p^  - 1),  where  p\  is 
given  in  (3)  , and  denotes  a disturbance  term,  one  may  adopt  the 
model 
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z = x!6  + £.  , 

3 3 i * 

and  estimate  6 by  least  squares.  F2(t),  F3(t),  and  other  transforma- 
tions may  be  adapted  in  an  analogous  way. 

MAXIMUM  LIKELIHOOD  ESTIMATION 

Estimation  of  the  model  in  (7)  by  maximum  likelihood  is  well  estab- 
lished for  the  logistic  cdf  case  (see  Cox,  1970,  pp.  87-91;  Hodges, 
1958;  Berkson,  1955),  and  also  for  the  normal  cdf  case  (see  Tobin, 

1955  and  1958  for  an  application  in  economics;  Fisher,  1935;  Finney, 

1947;  and  Cornfield  and  Mantel,  1950).  The  general  idea  is  outlined 
b e low . 

The  likelihood  function  for  the  model  in  (2)  is 


..,xG)  = 


G j yii 

n n [f(x’b)]  1J 

j=l  i=l  J 


For  problems  in  the  social  sciences  where  there  is  often  just  one 
observation  per  cell,  n.  = 1 for  all  j,  we  can  take  y.  E y , and 
rewrite  the  likelihood  function  as 


G y , 1-y 

L(y1»- • • .yGl- i,...,x  ) = n [f(x'b)]  J[i  - f(x!b)]  ^ . (ID 

j=i  2 3 

In  either  case,  (10)  or  (11)  may  be  maximized,  usually  by  numerical 
methods,  once  a suitable  F(*)  transformation  has  been  selected.  In  the 
remainder  of  this  report  (see  above,  p.  56  ff,  for  a justification)  we 

focus  upon  the  logistic  form;  hence,  we  give  results  only  for  that 
case. 

It  is  easy  to  check  that  for  one  observation  per  cell  if 
t*  = l±  x.y y t*  is  a sufficient  statistic  for  6 (given  vectors 
xl,,,,»xn)*  Note  that  t*  is  the  sum  of  those  x vectors  for  which  a 
response  was  obtained  (y^  = 1).  In  this  case,  the  MLE  of  6,  6,  must 
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satisfy  the  vectorial  equation 


n / -x'.fU  n 

Z ll+e  J)  x.  = t*  * E x y . . (12) 

3-1  V ! 3 j-l  3 3 


Also,  it  is  straightforward  to  check  that  log  L is  globally  concave, 
so  that  (12)  provides  an  absolute  maximum. ^ The  probability  of  a re- 
sponse at  stimulus  level  x is  then  estimated  by 

i-  (i  + .-x’s)-1 . 


?orj  a proof  in  the  more  general  case  of  a polytomous  dependent 

variable,  see  the  discussion  at  the  end  of  Section  A. 3. 
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A.3.  ONE  POLYTQMQUS  QUALITATIVE  VARIABLE 


In  many  situations,  the  response  variable  is  qualitative  but  there 
are  more  than  two  categories  into  which  it  may  fall.  Such  categorical 
variables  are  called  poly tomous . 

Let  y^>«»«»y  c*enote  Independent  (but  not  identically  distributed) 
observations  of  categorical  variables,  and  define 


P*3  5 P{y±  - a,)  , 


(13) 


where  the  response  variable  for  the  ith  subject,  y . , may  assume  Q 

Q 1 

values  a^,.*.,a  ; i = N;  then  £ p*,  = 1.  We  next  relate  the 

1 j=1  1J 

response  probabilities  to  stimulus  variables  by  means  of  the  standard- 
ized multivariate  logistic  distribution  cdf,  which  is  defined  by 


F( t . > ^n) 


n -t . 
1 4-  £ e 3 

3=1 


— °o  < t . < 00  . 

3 


(14) 


A symmetric  form  of  (14)  suggested  by  Mantel  (1966)  with  the  t.  appro- 
priately defined  in  terms  of  the  and  n related  to  Q,  is 


3 


pj 


Q 2 

I e 
k=l 


3 1 » • • • > Q 


(15) 


It  will  be  seen  in  (24)  below  that  this  symmetric  form  is  useful  for 
characterizing  several  correlated  polytomies  (multidimensional  contin- 
gency tables).  When  Q = 2,  the  model  in  (15)  may  be  reduced  to  that 


of  (7)  by  taking  p 


p,  and  z - z 


x1 6 . 


The  general  polytomous  model  of  (15)  with  the  z_.  expressed  as 
linear  functions  of  unknown  parameters  may  be  estimated  by  maximum 
likelihood,  as  was  done  in  Section  A. 2 for  the  dichotomous  model.  For 
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simplicity  of  notation  we  permit  the  to  be  multiple  observations  of 
the  same  group  of  objects;  but  we  ignore  such  information,  since  it  is 
not  needed  in  the  MLE  approach. 

Using  (13),  adopt  the  model 


p* 

Pij 


'ij 


Q 2 
Z e 
k=l 


ik 


i 1,,,.,N;  j *“  1,,,.,Q  , 


(16) 


and  take  z^  = Xi^j  ^or  ^xec*  x±:  s x 1*  and  y_.  a vector  of  coefficient 
weights.  Let  Z(N  x Q)  = (z„)  denote  that  Z is  an  N x Q matrix  with 


general  element  z 

J-J 

ij ^1 


.y  Then  if  Z(N  x Q)  = (z^)  , X(p  x N)  = (x^...,*^)  = 
(x^)t  r (p  x Q)  = (y^,,.,^^)  = (y^),  Z = XfT  and  T must  be  estimated 


for  given  X,  We  take  Z y . = 0 so  that  all  coefficients  will  be  uniquely 

1 3 

defined.  The  model  in  (16)  was  studied  by  Bock  (1970)  and  independently 
by  Press  (1972,  pp.  268-272).  Walker  and  Duncan  (1967)  proposed  a 
weighted  least  squares  solution,  which  is  equivalent  to  a maximum  like- 
lihood estimation  solution  in  large  samples.  The  maximum  likelihood 
estimation  solution  for  T is  given  below. 

Define  v,(  = 1 if  y = a,  and  let  v. . = 0 otherwise.  The  likeli- 

1 J iJ 

hood  function  is 


N Q v Q Q 

l = n n P*  i P*  = i,  i v..  - i . 

i=i  j=i  ij  j=i  ij  j=i  ij 


N 


Note  that  t 


. ~ Z x.  v, , is  sufficient  for  V given  X. 
j i=i  1 ^ 


The  result  of 


maximizing  L subject  to  Z y . =0  is  that  the  MLE  of  y , y # must 

1 3 l l 

satisfy  the  system  of  equations 


N 

Z 

i-1 


x;  y „ 
i 'l 


Q 

Z e 
k=l 


x y. 
i k 


x. 

l 


N 

Z 

i=l 


Xi  Vi£  * 


(17) 
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Q A 

and  Z y . = 0;  £ = 1, . . . ,Q.  (18) 

j=l 2  3 

The  solution  to  this  system  of  equations  must,  of  course,  be  obtained 
by  numerical  methods.  Whether  it  is  preferable  to  solve  (17)  and  (18) 
or  to  maximize  the  original  likelihood  function  depends  upon  the  parti- 
cular computer  software  available. 

Note  that  (17)  and  (18)  are  only  first  order  conditions  for  a maxi- 
mum, The  fact  that  they  guarantee  a maximum  is  seen  from  the  second 
order  conditions. 

Let  the  log  of  the  likelihood  be  given  as 

Q N 1“  Q xW  " 

L*  = log  L = Z t'y  - Z log  Z e 1 * . 

j=l  J J i-1  [k-1 

It  is  straightforward  to  show  that 


Z 6^  (m)(x  x!)  , 
i=l  1 11 


where 


Q 

Z exp{x!  (y,  - Y_)  } - 1 

•i-i  x 1C  m 

k=l 


2 

It  is  clear  by  inspection  that  6_^(m)  0,  and  since  (x^x^)  positive 

semi-definite  for  all  i,  L*  is  globally  concave;  so  a true  maximum 
corresponds  to  the  first  order  conditions. 

The  results  in  (17)  and  (18)  will  form  the  basis  for  parameter 
estimation  in  the  general,  multi-equation,  log-linear  model  (that  is, 
the  model  that  interrelates  many  categorical  variables  through  other 
explanatory  variables  and  interactions) . The  general  model  is  described 
in  Section  A. 4. 
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A.4.  SEVERAL  QUALITATIVE  POLYTQMQUS  VARIABLES 


GENERAL  MODEL 

In  earlier  sections  our  discussion  concerned  a single  qualitative 
variable,  for  which  the  probability  that  it  takes  on  a specified  value 
depends  on  certain  exogenous  or  explanatory  variables.  In  this  section 
we  explore  the  more  general  (multivariate)  case  of  any  number  of  jointly 
varying  polytomous  variables  and  exhibit  the  relation  between  the  log- 
linear  model  of  contingency  table  analysis  and  the  standard  logistic 
model  described  in  Section  A. 3.  After  we  define  the  basic  model,  we 
discuss  estimation  and  hypothesis  testing. 

In  this  analysis  it  is  not  the  value  of  a particular  qualitative 
variable,  be  it  dichotomous  or  polytomous,  that  is  important  but  the 
probability  that  the  variable  takes  on  a particular  value.  That  the 
probability  distribution  is  the  central  concept  in  the  usual  economet- 
ric context  is  often  obscured  by  the  emphasis  on  structural  and  reduced 
form  equations;  yet,  it  is  the  joint  probability  distribution  of  the 
unlagged  endogenous  variables,  and  various  conditional  distributions 
(or  certain  endogenous  variables  given  others),  that  are  at  the  heart 
of  the  analysis. 

Logit  analysis  of  several  related  qualitative  variables  by  weighted 
least  squares  was  proposed  by  Grizzle,  Starmer,  and  Koch  (1969)  and  by 
Grizzle  (1971),  Although  the  results  are  applicable  in  large  samples, 
they  ignore  the  probability  structure  of  the  errors,  and  therefore 
they  may  not  be  too  efficient  in  small  or  moderate  samples.  The  model 
described  below  uses  all  of  the  available  information  as  the  basis  for 
inference.  It  also  uses  a maximum  likelihood  estimation  procedure, 
which  may  be  regarded  as  desirable. 

Consider  the  general  multivariate  polytomy 3 which  is  often  referred 
to  as  a multidimensional  contingency  table . In  this  case  we  have  q 
variables,  y^,*..fy^,  which  may  take  on  any  one  of  I^,...,I  vcnordered 
alternative  values.  Whether  the  alternative  values  are  ordered  is 
quite  important  theoretically  but  perhaps  of  less  significance 
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practically . 


8 


The  total  number  of  possibilities  (cells)  is 


q 

Q - n 

k=l 


(19) 


for  example,  Q = 4 in  the  case  of  two  dichotomous  variables.  It  can 

easily  be  seen  that  as  long  as  the  problem  can  be  treated  as  one  in  which 

the  margins  are  not  fixed,  or  other  restrictions  introduced,  the  q- 

variate  polytomy  with  polytomies  of  order  I-,..*, I is  equivalent  to  a 

1 q 

univariate  polytomy  of  order  Q;  let  p.  . , with  i = 1,...,I  ;i 

l*  z *****  q ^ ^ 9 

be  the  probabilities  attaching  to  the  joint  events  that  y^ 

takes  on  the  i^th  value,  y ^ takes  on  the  i^th  value,  and  so  on;  then 
redefine 


(20) 


which  correspond  to  the  probabilities  of  the  events  that  some  polytomous 
variable  takes  on  the  first,  second,...,  or  Qth  value.  Arguing  in 
the  other  direction,  since  any  number  can  be  written  in  binary  form, 
any  polytomy  may  be  expressed  in  terms  of  a multivariate  dichotomy. 

This  equivalence  is  useful  in  the  analysis  of  jointly  dependent  quali- 
tative variables,  for  the  probabilities  for  a set  of  q jointly  depen- 
dent binary  variables  may  always  be  re-expressed  in  terms  of  a single 
polytomy  with  2^  possible  values,  and,  conversely,  any  polytomy  may  be 

reformulated  in  terms  of  a joint  occurrence  of  a certain  sequence  of 
9 

binary  variables. 


Analysis  of  the  ordered  case  may  be  quite  complex;  see  Cox  (1970, 
pp.  103-104).  Mantel  (1966,  p.  91),  however,  suggests  that  the  analysis 
not  be  constrained  by  the  ordering:  "But  there  is  no  need  in  applying 

the  model  to  know  the  ordering.  With  adequate  data  the  fitted  order- 
ing should  follow  the  true  parametric  ordering.  With  inadequate  data, 
inversions  may  occur,  but  the  same  kind  of  thing  is  true  in  ordinary 
regression  analysis  or  probit  analysis:  one  does  not  incorporate  into 

the  fitting  the  knowledge  that  the  slope  should  be  positive  but  depends 
rather  on  the  adequacy  of  the  data  to  bring  this  out." 

9 

In  general  one  will  need  some  additional  restrictions  on  the  proba- 
bilities of  the  joint  occurrence  of  these  binary  variables  since  the 
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Consider  now  the  probabilities  p.  , and  let  F(-)  denote  a 

1l,,,,,1q 

continuous  monotone  transformation  with  inverse  G(,)«  Let 


P 


i 


1 


(21) 


so  that 


P 


i 


V 


i 


q 


(22) 


for  = 1 , . . . , 1^; , , . ; iq  = 1,...,!^.  The  function  F(*)  is  taken  to  be 
the  natural  logarithm  (G ( • ) is  the  exponential  function)  so  that  if 
the  transformed  cell  probabilities  are  expressed  in  the  linear  form 


P. 

l 


1 


i 

q 


v + 0, 

V- 


i • 

q 


(23) 


for  i^  1, . * • , Ifj • . . ; iq  - l,..#fI  , the  cell  probabilities,  whose  q- 
dimensional  summation  must  be  unity,  assume  the  representation 


Pi  i 

l”#‘sq 


I 0. 

q i-i  > • • • » i 


(24) 


E . . . Z e 

i,=l  i =1 

i q 


This  is  the  symmetric  form  of  the  multivariate  logistic  function  dis- 
cussed in  Section  A. 3.  By  choosing  other  functions  for  F ( • ) alternative 
models  will  result,  such  as  the  multivariate  probit  model  (see  Ashford 
and  Sowden,  1970).  In  fact,  renaming  the  0s: 


polytomy  will  seldom  be  of  order  exactly  a power  of  2;  hence,  the  next 
higher  power  of  2 must  be  used  and  the  probabilities  of  occurrence  of 
certain  combinations  set  to  zero.  As  a result,  computational  difficul- 
ties may  arise  equivalent  to  those  that  occur  in  contingency  table  ana- 
lysis when  some  cells  are  empty.  Consequently,  in  cases  where  the 
polytomy  is  not  a power  of  2,  it  is  usually  not  preferable,  for  computa- 
tional reasons,  to  transform  it  into  a multivariate  dichotomy.  Moreover, 
such  transformations  may  obliterate  restrictions  inherent  in  the  problem — 
for  example,  those  imposed  when  certain  marginal  totals  are  fixed  as 
would  be  the  case  in  stratified  sampling. 
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Z1  °1 i;  ....  z _ 9i  • and  using  (20),  gives  the  univari- 

1*  * Q 

ate  polytomy  of  (15)  , 


Pj  " Q 

Z e k 
k=l 


j ~ 1»  • • » , Q • 


(15) 


The  parameters  (or  functions)  or  0^  ^ , however,  cannot  be 

chosen  fully  independently  because  they  are  not  identified  fully  by 
the  parameters  (or  functions)  pfc  or  p±  . , which  are  the  basic  para- 
meters of  the  problem.  To  see  this  simply  add  a constant,  c say,  to 
each  zk  m (15);  clearly  pk  is  unaltered;  thus  some  normalization  rule 
must  be  imposed,  such  as  = 0,  or,  more  conveniently  from  a theoreti- 
cal point  of  view  (see  also  Section  A. 3 where  the  analogous  argument 
is  presented) 


or,  equivalently, 


I 

q 

i e. 


V1 


= 0 . 


i =1 

q 


Li" 


(26) 


The  formulation  of  the  log-linear  model  of  contingency  tables  of 
Birch  (1963),  and  developed  further  by  Goodman  (1968,  1969,  1970,  1971a, 
1971b,  1972a,  1972b,  1972c)  and  others,  introduced  a convenient  para- 
meterization of  the  cell  probabilities  by  decomposing  the  9s  of  (24) 
into  main  effects  (functions  of  a single  subscript  ifc)  and  interaction 
effects  (functions  of  two  or  more  subscripts);  the  latter  are,  in  turn, 
distinguished  as  bivariate  interactions,  trivariate  interactions,  and 
so  forth.  All  of  these  are  treated  as  constants  by  Birch,  and  later 
by  Goodman  (1968,  1969,  1970,  1971a,  1971b,  1972a,  1972b,  1972c), 
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although  it  is  possible  to  generalize  the  analysis  by  allowing  them  to 
be  functions  of  exogenous  variables.  If  the  main  effects  are  denoted 
by  the  function 


ot^(i^)  , k 1, 

all  possible  bivariate  interactions  are  denoted  by  the  ) functions 


Sj^.(ij>i^)>  j < k l,...,q  , 


and  so  on;  then  the  log-linear  model  of  (24)  may  be  expressed  in  the 
form 


108  \ 1 ■ “ ■ \ i ’ “l(1l)  + -•  + W 


+ S12(il’i2)  + ”•  + Bq-l,q(1q-l’V 


• I 


+ “1,2 q(il’,,-’iq)  ’ 


(27) 


where  the  identifying  constraints  generally  used  are  given  by 


o^C*)  = a2^*^  = •••  = aq ^ ‘ = 0 * 


^12  C1!’  *>  612^*’  12>  ^q-1  ,q  ^ ^ ° ’ 


“l,2 q(*  *i2'***»iq)  “l,2,...,q(il,',"iq-l,")  ° ’ ^ 


and 


y = - log  Z ...  Z exp(0  . ) . 

i.  i 1 q 

1 q 
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A dot  denotes  summation  over  that  index.  It  is  easy  to  check  that  the 
conditions  of  (28)  imply  that  of  (26). 

The  model  in  (27)  and  (28)  may  be  used  to  study  the  q jointly  vary- 
ing qualitative  variables  in  terms  of  contributions  to  the  response  due 
to  main  effects  of  each  variable,  and  in  terms  of  interaction  effects 
among  variables,  much  as  these  variables  could  be  studied  using  the 
conventional  ANOVA  (if  the  response  variables  were  quantitative).  The 
results  in  either  model  are  nonspecific,  however,  in  that  typical  con- 
clusions from  such  an  analysis  are  that  there  are,  or_are  not,  inter- 
actions, and  that  certain  variables  interact  with  certain  other  ones. 

If  there  are  some  additional  explanatory  (exogenous)  variables  avail- 
able, a version  of  ANOCOVA  or  regression  may  be  attempted. 

Suppose  we  have  readily  available  observations  of  a vector  of 
exogenous  variables,  x (which  might  include  a constant  term),  and  for 
6 defined  in  (24)  we  take 


i » 

q 


(29) 


where  6*  is  expanded  in  terms  of  main  effects  and  interactions,  as  in 
(27).  For  example,  a^)  = x'a*^)  , B^tt^)  = x'g^U^ip , and 
so  on.  If  x is  in  fact  constant,  we  have  the  ANOVA  type  of  log-linear 
model.  If  x contains  bona  fide  explanatory  variables,  but  no  constant 
term,  we  have  a logistic  type  of  multivariate  regression.  If,  however, 
x contains  a "one"  as  one  of  its  elements  in  addition  to  bona  fide 
explanatory  variables,  an  ANOCOVA  type  of  log-linear  model  results. 

Some  of  the  elements  of  6*  might  be  taken  to  be  zero,  a priori , in  any 
case,  depending  upon  the  available  information  surrounding  the  problem. 
Below  we  detail  some  useful  properties  of  log-linear  and  logistic  models 
and  discuss  problems  of  inference  in  these  models. 

PROPERTIES  OF  THE  MULTIVARIATE  LOGISTIC  MODEL 

It  is  clear  that  (14)  and  (15)  are  completely  equivalent  forms  of 
the  same  distribution  since  (15)  may  be  rewritten  as 
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Q -(Zi-zk)  ’ 
1 + E e J 


a special  case  of  (14)  with  n = Q - 1.  Therefore,  the  properties  of 
(15)  or  (24)  may  be  studied  by  studying  those  of  (14) * 

Theorem  I 

All  marginal  distributions  associated  with  (14)  are  also  logistic. 

Proof:  Let  any  subset  of  t^  in  (14)  become  arbitrarily  large  and 

note  that  what  remains  is  still  logistic  of  some  dimension. 

Remark : A marginal  distribution  of  (14)  is  a marginal  of  a continu- 

ous vector  following  a logistic  cdf,  A marginal  distribution  of  (24), 
however,  is  a marginal  of  a discrete  distribution.  In  particular,  let 
z E (z^,...,2q)  and  p(z)  = [p^(z) , . . . ,p^(z) ] , where  p^(z)  denotes  the 
jth  cell  probability  for  a Q-dimensional  multinomial  distribution,  and 
each  cell  probability  is  permitted  to  depend  upon  a vector  z.  Then, 
if  Pj(z)  i-s  defined  as  in  (15),  p(z)  defines  the  cell  probability  vec- 
tor of  a standard  multinomial  distribution,  and  z indexes  the  distribu- 
tion. The  cell  probabilities  are  each,  in  turn,  related  to  the  cdf 
of  a multivariate  logistic  distribution  in  the  elements  of  z.  The 
marginals  of  (14)  must  be  logistic,  however,  for  the  term  multivariate 
logistic  to  have  any  conventional  meaning. 


Theorem  II 


All  conditional  distributions  associated  with  (14)  are  also  logistic. 


Proof:  Write  (14)  in  terms  of  two  subvectors  of  t = (t^)  = 

(t_,*..,t  j t i i , . . . , t ),  so  th  a t 
1 * r*  rfl*  n 


F(tr, 


n -t , 


1 + 1 e 
1 


Then,  because 


-1 
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■^(^1  > • * • > 


Gl(tl V'rU 5 t.)  • 


n 


and  from  Theorem  I,  c = G2 ^r+i*  • • • » tn>  = (l  + E e , 


G1(t1,.. •1tr|tr+1>. 


'V 


n -t 

1 + E e ^ 
r+1 


n -t  r -t 
J + E e J 


1 + I e 
r+1 


Equivalently, 


G1(t1,., 


, t:  t ,1 

r 1 r+1 


* * ’ * ,tn> 


-t  , 


1 + c 


(30) 


Note  that  since  (t^_^ , . . • , t^)  is  fixed,  c is  constant;  moreover,  (30) 
is  a multivariate  logistic  cdf  (non-s tandardized) , which  is  standard- 
ized by  subtracting  log  c from  every  t • 

j 

Theorem  III 

All  marginal  distributions  associated  with  (24)  are  in  multi- 
variate logistic  form  but  in  variables  different  from  those  of  the 
multivariate  distribution. 

Proof : Consider  the  univariate  (discrete)  marginal  distribution 

of  the  distribution  in  (24) : 
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where 


J 


* q 1 ...1 

Hl(±l')  = lo8  z ...  E e 1 q . 
i0=l  i =1 

2 q 

To  define  the  parameters  uniquely,  we  impose  the  condition  that 

h 

E = ^ ^or  every  a condition  that  can  be  satisfied  by  the 

V1 

H as  defined  above.  Clearly  p is  in  multivariate  logistic 

X1 

form  in  (H  ) and  p is  in  multivariate  logistic  form 

1 X1 * * * ’ * Xq 

in  the  0s.  In  the  same  way,  the  discrete  multivariate  marginal  proba- 
bilities are  also  expressible  in  multivariate  logistic  form,  for 
example, 
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H9  n(^o»« ••>!_) 
2.  . .q  Z q 


2 H (i0 , . . . ,i  ) 


2 ...  E e 

i9=l  i =1 
2 q 


2...q^2 


where 


i,  e. 

X I*  t • • 1 

^2  fl * * * * ’ = E e ^ 

Z...q  Z q , 

1 


For  uniqueness  of  the  parameters  we  require  that 


J ...  Z H ) = 0. 

i ± z...q  2 q 

2 q 


Theorem  IV 

The  conditional  probabilities  associated  with  (24)  have  a multi- 
variate logistic  representation  and  are  given  by 


exp  (\ o 

i|i  E exp  (d  \ * 

1 \ 1i»  * * • *■  q/ 


P-i  IT 


where  i denotes  a subvector  of  (.^,...,1  ),  and  T denotes  the  sub- 
vector of  (i^,...,i  ) whose  components  are  not  included  in  i_. 

Proof:  Define  the  marginal  distribution  of  the  discrete  variables 

corresponding  to  the  elements  of  i,  r_  * E p,  . . Then,  by 

definition,  i i 1 3 


pili 


pi  i 

l>.,,>  q 


exp  /0  \/E  E exp  /0  \ 

' l”*”  q//i  i \ 1l*,,*>1q/ 

z exp  (ei  pjnr^p  (ei  n~  * 

i V/ii  \ 1l»**,»q/ 


which  gives  the  required  result. 
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Remarks  : (1)  If  the  parameters  0.  are  assumed  to  be  func- 

11» * * * ’q 

tions  of  certain  explanatory  variables — for  example,  linear  functions — 
it  does  not  follow  that  the  marginal  probabilities  will  be  logistic  in 
functions  of  the  same  form  (this  result  is  implied  in  Theorem  III) . 

If  the  0s  are  linear  functions,  the  Hs  are  highly  nonlinear.  Nonethe- 
less, the  conditional  probabilities  are  logistic  in  functions  of  the 
same  form. 

(2)  Theorem  IV  is  used  in  the  section  on  hypothesis  testing  to 
define  the  conditional  odds  ratio  for  the  relative  levels  of  one  set 
of  variables,  given  another  set. 


Theorem  V 

The  qualitative  random  variables  y^,...,y^  defining  the  contingency 
table  whose  transformed  cell  probabilities  have  the  representation 
given  in  (29)  and  (30)  are  mutually  independent  if  and  only  if  all 
interaction  terms  vanish. 

Proof : This  result  may  be  shown  directly. 

If  all  interaction  terras  vanish,  it  is  straightforward  to  check 
that  the  marginal  probabilities  are  given  by 


w 

f ’ 

V1 


j l,..*,q  • 


But  from  (29),  with  no  interactions, 


\. 


- e 


a,(i.)  q 

J J = n r. 

j=l  J-l  xj 


q 

JI  e 


so  the  y_.  are  independent. 

Conversely,  suppose  the  y,  are  independent.  Then, 
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j = n r. 

i,,,,,xq  J-i  h 


where  the  = implies  an  identity  in  all  i . Equivalently, 


log  p. 


i i = z log  r . 

i q J-i  S 


Substituting  from  (27)  and  Theorem  III  gives 


u + W + = q u + z h (i  ) , 

j=i  J J q j=i  J j 


where  T(i1,...,i^)  denotes  a sum  of  terms,  each  of  which  depends  upon 
at  least  two  of  the  i_. . For  this  relation  to  be  an  identity  in  all 
1j  ’ must  be  zer°;  that  is,  independence  of  y1,...,y  implies  all 

the  interactions  must  vanish.  It  is  straightforward  to  check  that 
under  these  conditions 

q q 

1 H (i  ) = Z a (i  ) + p(l  - q) 
j=l  J J j=l  1 J 


identically  in  all  i . Note  that 


VV  WV 


Since  each  H^(i^)  depends  upon  only  a single  i^ , pr 


1 s 


is  expres- 


sible  in  general  multivariate  logistic  form  involving  only  main  effects 
(all  interaction  terms  are  zero).  Thus,  under  independence  the  model 
may  be  written 


log  P, 

I’’ 


q 

. = u + T,  a (i  ) . 

q j=i  J ^ 
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Double  Dichotomy 

The  results  of  this  section  may  be  illustrated  for  the  bivariate 
case  in  which  each  qualitative  variable  is  binary-that  is,  the  double 
dichotomy.  If  Yi  and  y£  can  each  assume  values  1 and  2,  the  joint 

probabilities  are  as  depicted  in  the  cells  of  the  following  contingency 
table. 


The  marginal  probabilities  are  shown  at  the  right  and  below.  The  para 

meters  of  the  log-linear  model  are  related  to  the  cell  probabilities 

through  (27)  and  (28),  which,  in  this  case,  if  we  take  p (i  ,i  ) = 

p.  . , reduce  to  1,2  1 2 

1’  2 

108  Pi1>i2  = y + W + «2(12>  + 612(il’i2)  * 

with  the  constraints  that  ct^.)  = o2(.)  = 0,  and  (^(-.i,,)  = 

= 0*  Defining 


ai  2 al(1)’  “2  2 a2 (1) * 8 = 812(1'1) 
for  simplicity  yields  the  following  equivalent  form:10 


log  P;q  = V + a1  + a2  + B , 

log  P12  = jj  + - a2  - 8 , 

loS  P21  = p " ai  + a2  " 3 , 

log  P22  = M - - a2  + 8 , 


(31) 


given^*  fT(4«?ati°n  °f  thiS  reSult  t0  ’ ■iichotomone  variables  is 
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and 


- y = log [ exp (a^  + a2  + g)  + exp^  - ct2  - 6) 

+ exp(-a1  + a2  - g)  + exp^o^  - o2  + g)]  . (32) 

Thus,  the  double  dichotomy  may  be  parameterized  in  terms  of  (a^,a2>g), 
or  equivalently,  in  terms  of,  say  (Pn,P12,  Pn) , with  p22  being  deter- 
mined from  pn  + p12  + p21  + p22  = 1.  — 

It  is  well  known,  and  straightforward  to  check,  that  the  condition 
for  independence  in  a double  dichotomy,  namely  P^j  = P^  Pj  , for 
^ ~ ^ an<^  3 — ^»2»  is  satisfied  if  and  only  if  the  cross  products 

of  the  cell  probabilities  are  equal — that  is,  P11P22  * p12p21'  Check~ 
mg  the  cross  product  condition  in  (31)  shows  that  it  is  satisfied  if 
and  only  if  there  is  no  interaction  term  in  the  log-linear  model  (see 
Theorem  III) . 

Now  define 


21  H a!  + a2  + g.  z2  5 “i  - a2  - 6 * 


z3  = -o^  + a2  - g,  z4  - -<*1  - a2  + B . 


Then  from  (31)  and  (32),  if  p.^  = p p2 


P12’  P3  = p21*  P4  ' p22’ 


the  log-linear  model  of  (31)  and  (32)  is  cast  into  the  form  of  the 
logistic  model 


4 

k=i  * 


j = 1.....4 


so  that  the  bivariate  dichotomy  now  corresponds  to  a univariate  poly— 
tomy  (see  (15))  with  four  unordered  categories. 

The  conditional  probabilities  promised  in  Theorem  IV  are  illus- 
trated for  the  double  dichotomy  by 
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p{y2  = 1 = 1}  = 


12 


(1,1) 


11 


Pi  + p2 


which  is  in  the  symmetric  logistic  form.  Equivalently, 


p{y2 


l 

-2 (a  +B)  ’ 

1 + e 


which  is  in  the  classical  (but  unstandardized)  logistic  cdf  form  in 

(<*2  + B). 

Suppose  there  are  q dichotomous  variables,  y^,...,y  whose  states 
are  designated  by  and  k2>  Then  if 


P{yl  ■ kiiy2  ■ y3  •••  ■ y,  ■ kl}  = fill : 


11 1 


-2t 


1 + e 


where 


= a1(k1) 


q 

+ z 

j=2 


Bi.j(ki> 


kl> 


q 

j=3 


Yl,2,j(kl* 


kl> 


+ 


“1,2 q<kl kl> 


and  the  a,  6,  are  defined  in  (27).  This  result  follows  immedi- 
ately from  Theorem  IV  when  the  0.  are  defined  as  in  (27), 

11»  * * • ’q 

To  understand  the  relationships  among  conditioning  variables  in  a 
double  dichotomy,  log-linear  or  logistic  model,  consider  first  the 
more  general  case  of  q qualitative  variables. 
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Let  y ~ ^2 yq^  » so  that 


r . Pi  aCi,j) 

><7!  - i|y  ■ J)  - --W-V 

Recall  that  a dot  denotes  a summation  over  an  index.  Then,  the  "odds 
ratio"  is  given  by 


P(yi  ' hjl ' J1  ”1 o(1i-» 

M7i-  i2|y-  J)  - PX ,<i2.J>  • 


Since  log  p 


1 5 P + 6.  the  log  of  the  odds  ratio  is 

x, . • • , q x j 


* i-Jy  = j> 

^i.iz  J)  5 108  ?(7i-  i2ly-  j)  ■ eii.J  ' 9i2.J 


(33) 


The  relation  in  (33)  shows  that  the  log-odds  is  always  a linear  func- 
tion of  the  9s*  Thus,  for  q * 2,  since 


Sii,J  ■ al(1l)  + “2<J>  + 612(11-J)  ' 


substitution  in  (33)  gives 


Riri2(j)  = ’ al(i2)]  + [612(il’j)  - ei2(12’j)] 


If  each  variable  is  binary,  as  in  the  double  dichototny  example  above, 


ai(ii)  + ai(±2)  = °»  s12(irj)  + e12a2,j)  = o , 


so  that 


Thus,  if  and  are  independent,  = ^ Ri  i ^ depends 

only  on  the  main  effect  for  y^.  In  the  absence  of  independence  the 
log-odds  depends  on  the  levels  of  the  conditional  variables  only 
through  the  interaction  term. 

The  result  in  (33)  provides  a means  of  analyzing  simultaneously 
determined  qualitative  variables,  in  particular  simultaneously  deter- 
mined dichotomous  variables,  in  terms  of  the  analogue  of  the  structural 
equations  in  standard  econometric  analysis.  This  analogue  is  just  the 
conditional  probability  function. 

In  the  more  general  multivariate  case,  the  conditional  probabilities 
depend  only  on  own  main  effects  and  on  those  interaction  effects  that 
involve  interactions  between  the  conditioned  and  conditioning  variables 
or  between  conditioning  variables.  This  is  seen  explicitly  below. 

From  Theorem  IV,  the  conditional  probability  of  any  subset  of  y_. 
given  the  levels  of  the  remaining  y ^ is  expressible  as 


P-r  IT  = 


exp  (0  ) 

* \9  ^ 


i|i  E exp  (0  , ) 

i 1l,##*’q 


Now  write  the  0 . , using  (27),  in  the  form 

If* • • • i Tq 


3 - <j>±  + <f>_ 

Xl***'*  q •=■  i 


where 


<|>  = Z a.(i  ) + (interactions  involving  elements  of  i)  , 

- jei  2 2 


4>_  = Z_a  (i.)  + (interactions  not  involving  elements  of  i) 
i jei  j j 
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Subs  ti  tuting  for  8.  . shows  that  drops  out  and  the  result 

Xl’*‘#,1q  i 

depends  only  on  main  effects  in  ji,  and  on  interactions  between  the  two 
sets  of  variates — that  is, 

exp  (<f>±)- 

p _ = (34) 

i|i  % exp  (4>  ) 

i - 

Note  that  may  involve  both  interactions  between  variables  in  i^  and 
in  and  interactions  between  variables  in  jL.  Interactions  between 
variables  only  in  i.  are  excluded,  however. 

It  is  sometimes  both  convenient  and  appropriate  to  ignore  inter- 
action effects  of  order  higher  than  two  and  to  assume  that  main  effects 
are  functions  of  the  exogenous  variables  but  interaction  effects  are 
not.  Then,  from  Theorem  IV,  it  is  possible  to  express  the  conditional 
odds  ratio  for  each  endogenous  variable  as  a linear  function  of  the 
exogenous  variables  and  those  interaction  terms  involving  the  endogen- 
ous variable  in  question  with  the  others,  which  are  treated  as  condi- 
tioning (see  Eq.  (34)).  Moreover,  when  all  endogenous  variables  are 
binary,  symmetric  forms  are  obtainable  by  treating  the  endogenous 
binary  variables  as  +1  or  -1  (see  Eq.  (41)).  The  choice  of  the  multi- 
variate logistic  distribution  in  the  above  models  may  be  given  a justi- 
fication in  terms  of  a physical  mechanism  by  extending  the  argument 
on  p,  14 . 

Let  E^,...,Eq  denote  a collection  of  mutually  exclusive  and  exhaus- 
tive events  (such  as  the  event  of  falling  into  each  of  the  Q cells  of 
a multi-dimensional  contingency  table)  and  let  x denote  a p-vector  with 
continuous  density  of  h(x|0),  for  some  parameter  matrix  0,  By  Bayes 
theorem, 

P {E,  }h(x|  E,  , 0) 

P{EJx}  = . 

E P{E.}h(x|E  ,9) 
j=l  J J 

Let  = P{Ej}  and  letcjf(x)  = N(0^,E)  if  is  true,  j = 


Then, 
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-1/2 (e^z  1ek  - 20^z_1x) 

P{Ek|x}  = — SS* ~ j- 

Q -1/2(0 'Z  V - 20'Z  x) 
Z q e 3 3 3 

J-l  J 


\ + bkx 

= e 

Q a,  + 9 

Z e J J 

d-i 


(35) 


for  an  obvious  set  of  weights  {a^  , b^}.  Since  renaming  the  8s  in  (24) 
and  making  them  linear  functions  of  some  exogenous  vector  x yields 
(35) , it  is  seen  that  the  basic  logistic  transformation  in  (24)  could 
have  arisen  in  a fundamental  way. 

INFERENCE  IN  THE  MULTIVARIATE  LOG-LINEAR/ LOGISTIC  MODEL 

The  multivariate  log-linear /logistic  models  discussed  earlier 
in  this  section  have  been  considered  so  far  only  from  a probabilistic, 
model  formulation  viewpoint.  We  now  consider  how  to  fit  the  models 
to  real  data  and  how  to  study  alternative  hypotheses  about  the  models 
on  the  basis  of  observed  evidence. 


Estimation 

It  is  well  known  (and  straightforward  to  check)  that  if 
Pl,2, . . . fq^i» # • • denotes  the  probability  of  falling  in  cell 

(iff •••»iq)  a dimensional  contingency  table,  and  if  N, 
q 

denotes  the  number  of  observations  falling  into  cell  (i  ,...,i  ),  the 

1 q 

MLEs  of  the  cell  probabilities  are  the  sample  cell  frequencies;  that 
is,  if  p denotes  the  MLE, 


N 


Pf  2 n ^1*  * * * 9 = 


i, , . . . ,i 

a 


N 


(36) 


where  N e ^ ...  E ^ denotes  the  total  number  of  observations 

in  the  contingency  table. 

»ow  suppose  the  cell  probabilities  are  expressed  „ exponential 
functions  of  the  main  effects  and  Interaction  effects  according  to  the 
log-linear  model.  The  parameters  of  the  log-linear  model  map  be  esti- 
mated by  Ml  by  solving  for  the  parameters  in  terms  of  the  cell  proba- 
bilities and  substituting  the  MLEs  of  the  cell  probabilities. 

In  the  case  of  the  double  dichotomy  discussed  above  (p.  77),  the 
parameter  relationships  are  as  given  in  (31) . In  ~cas"e,  for 
illustrative  purposes,  (31)  may  be  solved  simultaneously  to  obtain 


and 


-(1/4) log 


P11  P12 
P12  P22 


= (1/4)  log 


P11  P21 
P12  P22 


I 


6 - [l/ 4)  log 


P11  p22 
P12  P21 


(37) 


Therefore,  the  MLEs  are  given  by 


and 


vH^rr1, 

21  W22 


«2  - (1/4)  log  --11  “21 


Mli  N; 

N, „ N„„  » 


12  22 


e - (1/4)  log  - 


N11^22 

N 

12  1N21 


(38) 


It  will  always  be  possible  to  solve  for  the  MLEs  of  the  main 
effects  and  interactions  in  any  q-dimensional  polytomy  in  the  same  way 
All  relationships  will  always  be  linear  in  the  logs  of  the  cell 
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probabilities,  so  in  every  case^  the  result  will  be  completely  analog- 
ous to  (38) , This  situation,  where  all  parameters  of  the  log- linear 
model  are  present,  is  the  one  Goodman  calls  the  Saturated11  model  (see 
Goodman,  1970,  p.  253).  In  some  situations,  however,  it  may  be  desir- 
able to  set  some  of  the  parameters  equal  to  zero,  a priori > and  then 
estimate  the  remaining  parameters  of  this  "unsaturated"  model  by  MLE, 
Unfortunately,  except  for  some  simple  special  cases,  estimation  of  the 
unsaturated  model  is  generally  iterative  and  therefore  much  more  dif- 
ficult computationally  (see  Bishop,  1969,  for  a summary  of  various 
suggested  procedures).  For  example,  it  is  necessary  to  estimate  an 
unsaturated  model  when  one  desires  to  estimate  the  parameters  of  a log- 
linear  model  in  which  the  qualitative  variables  are  mutually  indepen- 
dent (so  that  by  Theorem  III,  all  interaction  parameters  must  be  taken 
to  be  zero);  or,  for  example,  when  interactions  of  order  higher  than 
two  are  to  be  ignored.  Estimation  of  parameters  in  a contingency  table, 
under  restricted  conditions  by  MLE,  was  begun  by  Bartlett  (1935)  (as 
suggested  by  R.  A.  Fisher)  for  the  case  of  a 2 x 2 x 2 table  with  fixed 
margins.  His  work  was  extended  by  Roy  and  Kastenbaum  (1956)  to  the 
case  of  an  r x s x t table  with  no  three-factor  interaction  term.  The 
work  of  Birch  (1963)  showed  that  marginal  totals  are  MLEs  of  their  ex- 
pectations (and  are  sufficient  statistics  for  these  parameters).  The 
"iterative  scaling11  technique  described  by  Bishop  (1969)  and  used  ex- 
tensively by  Goodman  (Goodman,  1970,  p.  237)  provides  a rapid  computa- 
tional algorithm  for  estimating  the  parameters  of  unsaturated  log- 
linear  models  of  the  ANOVA  type. 

If  the  log-linear/logistic  model  is  of  the  regression  type,  ML  may 
be  used  directly  to  estimate  the  coefficients  of  the  explanatory  vari- 
ables, in  addition  to  the  effects  themselves.  The  ANOVA  parameter- 
ization is  a special  case  of  the  general  model  and  can  be  studied  by 


^Difficulties  arise  in  the  event  of  empty  cells  (see,  for  example, 
Fienberg,  1972,  for  a discussion  of  this  case).  In  the  unsaturated 
log-linear  model,  it  may  be  possible  to  identify  some  of  the  parameters 
even  when  some  of  the  cells  are  empty. 
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placing  the  ANOVA  type  of  log-linear  model  into  the  regression  type  of 
log-linear  model  formulation. 

Suppose  a q- variate  polytomy  with  each  of  the  qualitative  variables 
related  to  a p-vector  of  exogenous  variables,  x.  Let  p*^  = P{ith  subject 
falls  into  cell  j},  i = 1,...,N,  and  j = 1,..,,Q,  with 

q 

Q = II  I.  from  (19)  . The  logistic  model  gives 
k=l 


i - 1, . , , ,N;  j — 1 f-g-m » , Q . 


Now  with  z„  = x^y j > for  some  observed  p-vectors  of  exogenous  variables, 
x^,...,x^,  the  MLEs  of  the  y are  found  as  the  solutions  of  (17)  and 
(18),  as  in  Section  A. 3, 

To  fit  the  logistic  model  in  the  manner  shown  above  requires  only 
the  logistic  transformation  assumption.  However,  if  we  impose  the 
ANOVA  type  of  structure  of  (27)  , the  terms  in  the  exponents  in  (16) 
will  be  expressed  as  main  effects  and  interaction  terms,  each  of  which 
may  be  expressed  as  a linear  function  of  exogenous  variables  (see  the 
formulation  in  (29)).  To  facilitate  understanding  consider  the  double 
dichotomy  [example  (p.  77).  The  model  may  be  parameterized  as  in  (31) 
and  (32)  . Now  suppose 

= x(l),y(l),  a 2 « x(2)fy(2),  and  6 = x(3)  'y(3)  » (39) 

where  x(l) , x(2) , x(3)  are  each  vectors  of  exogenous  variables,  and 
y(l)  , y(2),  and  y(3)  are  vectors  of  weights.  Then,  for  example. 


log  P11  " y + x(l)fY(l)  + x(2)Ty(2)  + x(3)fy(3)  , 


log  pn  = + x'y  * 


(40) 
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where  x'  = (x(l)\  x(2)\  x(3)'),  and  y'  = (y(l)',  y(2)\  y(3)’).  When 
(39)  is  expressed  with  appropriate  subscripts  for  observed  data,  it  is 
equivalent  to  (16)  with  z = x^y^ . It  is  merely  a matter  of  setting 

some  of  the  elements  of  the  y vector  equal  to  zero  a priori  to  complete 

1 TO " •' 

the  comparison.  The  advantage  of  the  latter  formulation  is  that  spe- 
cific main  effects  or  interaction  terms  may  be  "explained"  in  terms 
of  exogenous  variables  believed  to  be  appropriate,  a priori.  In  parti- 
cular, if  x^  is  the  value  of  x for  the  ith  subject,  and  is  the  co- 
efficient vector  corresponding  to  cell  (1,1) , 

3 

log  = P + x’y1  = u + I x±(j)  tY1(j)  , i = 1, . . . ,N  , 


in  exact  agreement  with  (16) . 

The  approach  used  above  for  the  double  dichotomy  may  of  course  be 
generalized  to  any  multidimensional  model.  In  particular,  the  approach 
is  used  for  the  multivariate  case  with  only  bivariate  non-zero  inter- 
action effects  in  a computer  program  developed  especially  for  implemen- 
tation of  these  results. 

In  situations  where  the  main  effects  and  interactions  have  been  es- 
timated and  it  is  desired  to  estimate  the  cell  probabilities,  the  latter 
may  easily  be  reconstructed  from  the  former  by  substituting  the  esti- 
mated parameters  into  the  right  hand  side  of  (27).  In  problems  involv- 
ing endogenous  variables  that  are  all  dichotomous,  it  is  sometimes 
convenient  to  denote  the  states  by  one  and  minus  one  and  then  define 
all  main  effects  and  interactions  in  terms  of  the  values  in  the  "one" 
states.  Suppose  the  cell  probabilities  are  defined  as  in  (27),  subject 
to  the  constraints  in  (28) , and 

0*1  = o*i(l)  , 0*2  - 0*2  (1)  9 • • • y aq  “ o*^(l)  * 


*1,2  " 61,2(1)’  31,3  ” 3l,3(1),,,*’eq-l,q 


q-i*q 


(D; 


^Elements  of  y could  be  set  equal  to  any  pre-assigned  values  other 
than  zero  as  well,  which  might  be  useful  in  some  circumstances. 
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l|2>*»*,q 


” W1  2 a 


...i) 


Then,  it  is  straightforward  to  check  that  any  cell  probability  may  be 
expressed  as 


log 


Pi  i 

l**"*q 


= P + 


Ul°l  + 


U2a2 


u a 

q q 


+ uiU2B1i2  + u1u3B1i3 


1 u 


tU  P - 

q— i q q-i.q 


+ . • . 


^1  * • • U L0-  ry  y 

12  9l»2,,,.,q 


(41) 


where  u_,  - 4*1,  -1  for  all  j = l,...,q,  depending  on  the  state  expressed 
by  Uj  . 

Hypothesis  Testing 

A likelihood-ratio  test  may  be  used  to  test  any  hypotheses  about 
the  model,  for  example,  the  hypothesis  of  independence.  In  general, 
the  likelihood  ratio,  X,  is  the  ratio  of  the  value  of  the  likelihood 
function  maximized  under  whatever  constraints  are  embodied  in  the  hypo- 
thesis being  tested  to  the  value  maximized  under  no  constraints  except, 
of  course,  those  implicit  in  the  general  model.  Although  in  small 
samples  the  distribution  of  X is  very  complicated,  in  large  samples 
the  quantity  -2  log  X is  distributed  as  Chi  square  with  as  many  degrees 
of  freedom  as  there  are  independent  restrictions  embodied  in  the  hypo- 
thesis being  tested,  relative  to  its  alternative.  Hypothesis  tests 
of  relationships  in  a contingency  table  are  not  new  and  can  be  traced 
back  to  the  work  of  Karl  Pearson  (1900).  It  was  not  until  the  recent 
work  on  the  log-linear  model  of  Goodman  (1969)  for  three-way  tables, 
and  his  extensions  to  m-way  tables  (in  Goodman,  1970),  that  satisfac- 
tory hierarchical  methods  were  proposed  for  partitioning  Chi  square 
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statistics  based  upon  the  likelihood  ratio  criterion.  Goodman  suggests 
we  might  start  by  testing  that  the  highest  order  interactions  in  a 
log-linear  model  are  zero.  If  the  hypothesis  is  accepted,  we  might 
test  that  the  next  lower  order  interaction  terms  are  zero  given  that  the 
higher  order  interaction  terms  are  zero,  and  so  on.  Proceeding  in  this 
way,  the  likelihood  ratio  test  statistics,  A,  are  independent,  and 
asymptotically , -2  log  A is  distributed  as  Chi  square. 

"Causal  relationships"  and  associations  among  the  qualitative  vari- 
ables in  a contingency  table  are  studied  using  the  log-linear  model 
by  estimating  the  parameters  of  the  log  of  the  odds  ratio  for  a given 
qualitative  variable  expressed  as  a linear  function  of  the  effects,  as 
in  (27)  . The  details  of  this  type  of  approach  may  be  found  in  Bishop 
(1969),  and  Goodman  (1971a,  1972a,  1972b,  1972c).  This  type  of  analy- 
sis is  appropriate  when  the  only  information  available  is  a cross  tabu- 
lation of  qualitative  variables.  However,  if  there  are  some  continuous, 
or  discrete  but  ordered,  exogenous  variables  available  that  can  be 
related  to  individual  subjects,  they  should  be  introduced  where  appro- 
priate, and  hypotheses  concerning  their  coefficients  may  be  tested  by 
likelihood  ratio  procedures . 

Testing  whether  a particular  exogenous  variable  or  even  a group 
of  such  variables  is  a significant  determinant  of  the  probabilities 
can  be  carried  out  by  the  following  likelihood  ratio  test:  Replace 

the  parameter  values  by  zero;  maximize  subject  to  any  additional  con- 
straints, and  compare  the  result  with  the  result  unconstrained  except 
by  the  maintained  constraints  of  the  model;  -2  times  the  difference 
is  distributed  asymptotically  as  Chi  square  with  as  many  degrees  of 
freedom  as  coefficients  set  to  zero. 
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studying  the  distribution  of  - y^  - y^  + y^f  which  is  normal  in 
large  samples  since  is  a MLE.  The  asymptotic  variances  and  co- 
variances  of  the  Yj  are  easily  obtained  from  the  inverse  of  the  in- 
formation matrix  based  upon  the  likelihood  parameterized  in  terms  of 

^ V 

More  generally,  we  may  want  to  test  hypotheses  of  the  form:  " “ 

H:  LTM  = 0 , 

where  F _ (y^»»».»Yq)*  p x Q is  the  matrix  of  model  parameters  and  L 
and  M are  arbitrary  preassigned  matrices.  Likelihood  ratio  tests  are 
still  applicable  and  LrM  is  normally  distributed  in  large  samples. 

Hypotheses  may  be  tested  using  the  original  model  or  in  terms  of 
a conditional  probability  model  involving  only  the  parameters  of 
interest.  Since  certain  endogenous  variables  or  levels  of  these  vari- 
ables are  held  fixed  in  the  conditional  probability  model  (and  are 
therefore  treated  as  exogenous),  estimators  based  on  this  model  will 
not  be  the  same  as  those  obtained  from  the  unconstrained  (unconditioned, 
jointly  estimated)  model.  We  term  the  estimators  obtained  from  the 
conditional  probability  model  ^conditional  estimators,  and  those  from 
the  unconstrained  model  fuZZ-information  estimators. 


CONDITIONAL  PROBABILITIES  AND  "CONDITIONAL  ESTIMATORS" 

In  many  situations  we  are  concerned  with  the  estimation  of  oondi- 
tionaZ  probabilities  and  tests  of  hypotheses  about  those  conditional 
probabilities.  In  the  context  of  jointly  dependent  qualitative  vari- 
ables , the  conditional  probability  function  corresponds  to  the  struo- 
turaZ  equation  of  the  simultaneous  equations  approach  (of  traditional 
econometrics)  to  the  analysis  of  continuously  variable  jointly  depen- 
dent endogenous  variables.  For  example,  in  the  analysis  of  prices  and 
quantities  jointly  determined  by  supply  and  demand,  our  interest  fre- 
quently centers  on  the  elasticity  of  demand,  which  expresses  the 
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variation  of  the  quantity  demanded  conditional  on  variations  in  price. 
This  section  explores  more  fully  the  form  of  the  conditional  probabil- 
ity functions  for  several  jointly  dependent  dichotomous  variables;  in 
the  process,  we  find  an  interpretation  of  what  econometricians  have 
frequently  done  when  confronted  with  problems  involving  multiple  quali- 
tative endogenous  variables. 

Consider,  for  example,  the  trivariate  dichotomous  case  of  the  log- 
linear  model,  (27)  and  (28)  above.  (The  main  and  interaction  effects 
may  be  functions  of  explanatory  variables,  but  such  dependencies  are 
here  suppressed.)  Let  the  indices  i^,  i ^ » i^  take  on  the  convenient 
values  1 for  occurrence,  and  0 for  nonoccurrence.  In  the  trivariate 
dichotomy,  there  are  8-1=7  distinct  probabilities  corresponding 
to  the  three  main  effects: 

^1 * a2 * a3  * 

the  three  bivariate  interaction  effects: 

312’  3 * 13 * 323  5 


and  the  one  trivariate  interaction  effect: 


13 

|_  Henry  Schultz  (1938)  and  others  of  his  day  did  indeed  attempt  to 
estimate  demand  equations  directly  without  reference,  or  with  only 
limited  reference,  to  the  simultaneous  systems  in  which  these  relations 
are  embedded.  The  development  of  full  and  limited  information  maximum 
likelihood  methods  of  estimation  in  the  1940s  permitted  a more  sophisti- 
cated approach,  corresponding  to  estimation  based  on  the  joint  proba- 
bility function  of  the  jointly  dependent  endogenous  variables.  Here 
we  suggest  an  analogue  to  what  Schultz  and  others  did  in  the  case  of 
demand  functions  for  the  estimation  of  conditional  probability  func- 
tions for  jointly  dependent  qualitative  variables.  Although  such 
methods  are  known  to  be  inappropriate  in  the  case  of  continuously  vari- 
able data,  they  are  common  in  what  few  analyses  of  qualitative  data 
econometricians  have  undertaken  to  date. 
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Y123  ’ 

Given  the  restrictions  (28),  all  of  the  joint  probabilities  may  be 
expressed  in  terms  of  these  seven  parameters  (see  also  Eq.  (41)): 


log  pm  - V - Oj  + «2  + + e12  + 613  + iS23  + y123 
iog  P110  - u - + «2  - «3  + 612  - B13  - 623  - y123 
log  P100  - u - - P2  - «3  - eu  - S13  + e23  + y123 


log 

P000 

“ U = 

"ai 

a2 

“3 

+ 612  + 

g13 

+ g23 

~ y123 

log 

P001 

- y *= 

■°i 

- a2  + 

a3 

+ 612- 

ei3 

623 

+ Y123 

log 

P011 

- y * 

-“i 

+ a2  + 

a3 

ei2 

6 13 

+ 623 

~ Y123 

log 

P101 

- y « 

ai  ' 

- a2  + a3  - 

- 3i2  + 6 

13  ' 

623  ' 

' Y123 

log 

P010 

- y = 

“i 

+ a2  - 

a3 

" ei2  + 

ei3 

623 

+ Y123 

(42) 


The  parameter  p is  defined  so  that 


Pi  < < 3 • 

il,i2,i3=0  123 


Using  the  result  of  Theorem  IV  the  conditional  probabilities  may 
also  be  expressed  in  an  instructive  way.  Take,  for  example,  the  proba- 
bility of  the  occurrence  of  the  first  event  conditional  on  the  second 
and  third  events  (see  Eq.  (34)): 

log  - Uj*  - + ei2  + s13  + y123 

i»s  Pt.id2-1.i3-0)  - Ul*  - »i  + e12  - e13  - y123 

log  P^fVO.ij.l)  - Pi*  - «i  - 612  + ei3  - y123 


108  Pi3.id2-0.i3-0)  - Pi*  - „i  - e12  - 613  + 


123* 


(43) 


The  parameter  p1*  is  given  by 


uj  = - log  |exp(a1 


+ 6 


1,2  Y1,2,3J 


The  conditional  probabilities  p±  =1(i1»i3)  and  P±  ^(i^,^)  ^ be 
defined  similarly.  ^ ^ 

Equations  (43)  may  be  rewritten  in  an  interesting  and  useful  form 
by  introducing  new  variables  u^  u2,  u3>  which  take  on  the  values  of 
+1  or  -1  depending  on  the  occurrence  of  the  first,  second,  or  third 
event  (see  Eq.  (41)).  (Earlier,  we  introduced  the  variables  y^,  y^ , 
y3»  taking  on  the  values  1 or  0;  the  u represent  a simple  rescaling 
of  such  variables  more  convenient  for  our  present  purpose.)"^  Equa- 
tions (43)  may  be  re-expressed  in  terms  of  a univariate  logistic 
formulation: 


e^°‘l+612+B13+Y123} 

11_1  2 3 {al+612+613+Y123}  , “{al+6l2+ei3^123} 


+ e 


1 + e 


-2{a1+B12+613+Y123} 


Pn.  =1  (i9=l,i  =0)  = — T , 


1 + e 


1 K12  13  123 


►(44) 


P,  ,(i2=0,i  =1)  = ~j — y , 

1 1 . , 2{“l  B12+ei3~Y123} 

1 + e 


P-t  =1  (i.,=0,i,=0)  = 
1 


-*3-"  - - -«c1-Bl2-ei3-^123>  > 

1 t e 


so  that,  in  compact  form, 


The  scaling  of  a qualitative  variable  is  completely  arbitrary  in 
a theoretical  sense.  See  p.  88  for  a previous  use  of  their  scaling  de- 
vice. 
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P-J  =1  ^9 


V1 


1 + e 


2{a1+312u2+ei3u3+Y123u2u3} 


(45) 


Equation  (45)  suggests  an  analogue  to  the  ordinary  least  squares 
estimation  of  individual  structural  equations  within  a system  of  simul- 
taneous economic  relationships.  We  know  that  such  estimates  have 
undesirable  statistical  properties  and  are  inappropriate  in  the  sense 
that  they  do  not  reflect  the  simultaneity  of  the  system  in  which  the 
individual  structural  relation  is  embedded.  With^each  value  of  y^n» 
n « 1, reflecting  the  occurrence  (1)  or  nonoccurrence  (0)  of 
the  first  event,  are  values  of  the  variables  u^n  and  u^n»  reflecting 
the  associated  occurrence  (+1)  or  nonoccurrence  (-1)  of  the  second 
and  third  events.  The  "likelihood  function"  associated  with  the  se- 


In*  ~2n*  a3n' 

u_  as  if  they  were  exogenous , is: 


quence  of  observations  (y^,  u0„ , u^)  , n - 1,...,N,  treating  u^n  and 
*3n 


L*  ^“l’612,B13’Y123^ 


where 


N 

n 
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yln 

1 
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rt 
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rt 
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rt 

t -t. 

n . n 

Le  + e J 

e + e 

(46) 


= + 


12 


u2n  + 


613  u3n  + t123  u2n  u3n 


Maximizing  L*  with  respect  to  the  parameters  a^,  ^^2 9 ^13*  ^123*  ^^dh 
appear  in  the  conditional  likelihood  function  for  the  first  event, 
yields  an  interesting  kind  of  estimator  of  these  parameters  (to  be  con- 
sistent the  endogenous  qualitative  variables  y % and  y ^ should  be 
rescaled  as  u0  and  u„9  to  take  on  the  values  -hi  or  - 1 ♦ Such  esti- 

O O 

mators  are  not  appropriate  estimators  if  one  really  believes  the 
trivariate  dichotomous  model  for  the  qualitative  variables  y^,  y y 3* 
but  just  as  ordinary  least  squares  estimates  of  the  parameters  in  a 
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structural  equation  may  not  be  far  off  the  mark  compared  with  full  in- 
formation maximum  likelihood  estimators,  so  the  above  estimators  may 
not  be  "bad"  estimators  of  a^,  £3.3  anc*  ^±23*  Moreover>  they  may 

be  a great  deal  cheaper  to  compute.  We  call  estimators  based  on  maxi- 
mization of  L*  conditional  estimators . Application  of  this  method  to 
each  of  the  three  conditional  probability  functions  (for  example,  by 
using  a univariate  dichotomous  program),  yields  one  estimate  each  of 
a^,  a^,  and  a^,  but  two  estimates  each  of  ^13*  and  ^23*  three 

distinct  estimates  of  Y^.23* 

In  the  computer  program  developed  for  this  problem,  we  restrict 
our  model  to  a simple  and  convenient  form  by  assuming  that:  (a)  in- 

teraction effects  of  order  higher  than  two  are  absent;  (b)  bivariate 
interaction  effects  are  constant,  independent  of  any  explanatory  ex- 
ogenous variables;  and  (c)  main  effects  are  linear  functions  of  any 
explanatory  exogenous  variables.  Under  these  simplifying  assumptions, 
examination  of  Eqs.  (42)  and  (43)  reveals  that  the  exponents  (alterna- 
tively half  the  logarithmic  odds  ratios)  in  each  case  may  be  expressed 
as  linear  functions  of  the  exogenous  variables  and  the  scaled  variables 
u^,  u^,  Uy  Thus,  these  simplifying  assumptions  correspond  to  the  as- 
sumption of  linear  structural  equations  in  a simultaneous  equations 

_ 15 

system. 

The  applicability  of  the  methodology  described  in  this  appendix  is 
illustrated  in  Nerlove  and  Press,  1974,  Section  5.  It  is  shown  there, 
by  means  of  a variety  of  empirical  economic  applications,  that  the 
above  procedures  are  required  in  many  situations  where  alternative 
procedures  give  misleading,  inaccurate,  or  incorrect  results.  In  par- 
ticular, in  that  report  we  compare  the  results  obtained  for  dichotomous 
dependent  variables  using  the  linear  probability  with  those  obtained 
using  maximum-likelihood  methods,  and  we  compare  the  estimates  of  the 
joint  probabilities  of  several  dichotomous  variables  with  those  ob- 
tained by  treating  all  but  one  of  the  jointly  dependent  dichotomous 

15 Unlike  the  latter,  however,  the  qualitative  variable  case  does 
not,  in  general,  present  identification  problems  under  these  simplify- 
ing assumptions. 
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variables  as  if  they  were  exogenous  (conditional  estimators).  Empirical 
application  of  this  model  has  not  yet  been  made  in  the  context  of  mea- 
suring technological  change. 
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